Object tracking processing method and apparatus, device, and medium

Abstract

The present invention provides an object tracking processing method and apparatus, a device, and a medium. The processing method comprises: respectively preprocessing a first-frame image and a current image to generate a corresponding first-frame initial image and a corresponding current initial image; performing feature extraction processing on the first-frame initial image and the current initial image by means of a weight-adaptive fusion encoder to generate input feature data; performing deep-level fusion processing on the first-frame initial image and the current initial image; performing feature extraction processing by means of the weight-adaptive fusion encoder to generate deep feature data; performing attention-weighted fusion processing on the input feature data and the deep feature data by means of the weight-adaptive fusion encoder to generate target feature tensor data; and processing the target feature tensor data by means of a fully convolutional neural network to generate a target image. The object tracking processing method and apparatus, the device, and the medium provided by the present invention improve object tracking performance.

Description

A target tracking processing method, apparatus, device, and medium Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method, apparatus, device, and medium for target tracking. Background Technology

[0002] With the continuous development of robotics technology, the demand for robots in various dynamic and complex environments is gradually increasing, especially in fields such as autonomous driving, industrial automation, and intelligent monitoring. The environmental perception capability of robots is crucial. Robot vision perception systems acquire environmental information through image sensors and use computer vision technology to detect, locate, and track targets.

[0003] However, traditional visual perception systems face several key challenges, particularly in situations involving complex backgrounds, dynamic scale changes, low lighting, and occlusion. Traditional image processing methods often perform poorly in these conditions, impacting the robot's localization and decision-making capabilities. Complex backgrounds make it difficult for traditional image processing methods to effectively separate targets from the background, leading to both false positives and false negatives. Therefore, areas for improvement exist. Summary of the Invention

[0004] The purpose of this invention is to provide a target tracking processing method, apparatus, device, and medium that can improve the target tracking effect.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0006] This invention provides a target tracking processing method, comprising:

[0007] The first frame image and the current image are preprocessed separately to extract the target region in the image and generate the corresponding first frame initial image and current initial image;

[0008] The first initial image and the current initial image are processed by a weighted adaptive fusion encoder to extract features and generate input feature data.

[0009] The first initial image and the current initial image are subjected to deep fusion processing; and the processed image is subjected to feature extraction processing by the weighted adaptive fusion encoder to generate deep feature data.

[0010] The input feature data and the deep feature data are fused by the weighted adaptive fusion encoder to generate target feature tensor data.

[0011] The target image is generated by processing the target feature tensor data using a fully convolutional neural network.

[0012] In one embodiment of the present invention, the step of performing depth fusion processing on the first initial image and the current initial image includes:

[0013] Depth estimation processing is performed on the first initial image and the current initial image respectively to generate the corresponding first-frame depth image and current depth image;

[0014] Based on the center depth value of the previous frame image, the first frame depth image and the current depth image are normalized respectively to generate the corresponding first frame depth-enhanced image and current depth-enhanced image.

[0015] In one embodiment of the present invention, the step of performing attention weight fusion processing on the input feature data and the deep feature data through the weight adaptive fusion encoder to generate target feature tensor data includes:

[0016] The input feature data and the depth feature data are normalized by the initial normalization layer of the weight adaptive fusion encoder to generate corresponding input normalized data and depth normalized data.

[0017] The linear layer of the weighted adaptive fusion encoder processes the input normalized data and the depth normalized data respectively, and calculates the corresponding input attention parameters and depth attention parameters.

[0018] The input attention parameters and the depth attention parameters are processed by the softmax layer of the weight adaptive fusion encoder to generate corresponding input attention weights and depth attention weights.

[0019] Calculate the inner product of the input attention weights and the median vector matrix of the input attention parameters to generate input summary data; calculate the inner product of the deep attention weights and the median vector matrix of the deep attention parameters to generate deep summary data;

[0020] The input attention weights and deep attention weights are processed by the attention weight fusion layer of the weight adaptive fusion encoder to generate corresponding hybrid attention weight values;

[0021] The feature-weight fusion layer of the weight adaptive fusion encoder processes the hybrid attention weight values, the input summary data, and the input feature data to generate corresponding input merged feature data; the feature-weight fusion layer processes the hybrid attention weight values, the depth summary data, and the depth feature data to generate corresponding depth merged feature data.

[0022] The input merged feature data and the depth merged feature data are processed by the output normalization layer of the weight adaptive fusion encoder, and the processing results are input into the fully connected layer of the weight adaptive fusion encoder for further processing to generate corresponding input feature tensor data and depth feature tensor data.

[0023] The input feature tensor data and the depth feature tensor data are subjected to multiple attention weight fusion processes by the weight adaptive fusion encoder to generate the target feature tensor data.

[0024] In one embodiment of the present invention, the hybrid attention weight value W AMF , represented as: W AMF =A F +A D , where A F A represents the input attention weights. D This represents the weights for deep attention.

[0025] In one embodiment of the present invention, the input merged feature data B F , represented as: B F =F+(A′) F ⊙W AMF ), where F represents the input feature data, A′ F This represents the input summary data, and ⊙ represents element-wise multiplication; the deep merging feature data B D , represented as: B D =D+(A′) D ⊙W AMF ), where D represents depth feature data, A′ D This is represented as deeply summarized data.

[0026] In one embodiment of the present invention, the step of generating target feature tensor data by performing multiple attention weight fusion processes on the input feature tensor data and the depth feature tensor data through the weight adaptive fusion encoder includes:

[0027] The input feature tensor data and the depth feature tensor data are subjected to multiple attention weight fusion processes by the weight adaptive fusion encoder, and the target feature tensor data is obtained based on the final output input feature tensor data.

[0028] In one embodiment of the present invention, the step of processing the target feature tensor data using a fully convolutional neural network to generate a target image includes:

[0029] The target feature tensor data is processed by a fully convolutional neural network to generate the corresponding target classification score map, normalization size parameter, and local offset matrix.

[0030] The initial position of the target is determined by locating the peak value in the target classification score map;

[0031] The initial position is corrected based on the local offset matrix to determine the final position of the target;

[0032] The set bounding box is scaled according to the normalized size parameter, and the final position is encapsulated by the scaled bounding box to generate the target image.

[0033] The present invention also provides a target tracking processing apparatus, comprising:

[0034] The preprocessing module is used to preprocess the first frame image and the current image respectively to extract the target region in the image and generate the corresponding first frame initial image and current initial image;

[0035] The first extraction module is used to perform feature extraction processing on the first frame initial image and the current initial image to generate input feature data;

[0036] The second extraction module is used to perform depth fusion processing on the first frame initial image and the current initial image; and to perform feature extraction processing on the processed image to generate depth feature data;

[0037] The deep fusion module is used to perform attention weight fusion processing on the input feature data and the deep feature data to generate target feature tensor data; and

[0038] The image output module is used to process the target feature tensor data to generate the target image.

[0039] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the target tracking processing method described above.

[0040] The present invention also provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the target tracking processing method.

[0041] As described above, this invention provides a method, apparatus, device, and medium for target tracking. By enhancing depth information and adaptively fusing multimodal information, combined with an attention mechanism, it significantly improves the target tracking accuracy and robustness of robots in complex scenarios. This method not only effectively addresses common problems such as dynamic scale changes and blurred targets, but also maintains high target recognition and tracking stability in complex environments. Its broad application prospects, particularly in autonomous driving, industrial automation, and intelligent monitoring, provide crucial support for future technological development.

[0042] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 is a flowchart of a target tracking processing method according to an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of a weighted adaptive fusion encoder in one embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of a target tracking processing device according to an embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0048] In the diagram: 100, preprocessing module; 200, first extraction module; 300, second extraction module; 400, deep fusion module; 500, image output module; 1, electronic device; 12, memory; 13, processor. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Referring to Figure 1, this invention provides a target tracking processing method that can be applied to robots to enable them to accurately track targets in complex scenarios. The processing method may include the following steps:

[0051] Step S10: Preprocess the first frame image and the current image respectively to extract the target region in the image and generate the corresponding first frame initial image and current initial image;

[0052] Step S20: Perform feature extraction processing on the first initial image and the current initial image using a weighted adaptive fusion encoder to generate input feature data;

[0053] Step S30: Perform depth fusion processing on the first initial image and the current initial image; and perform feature extraction processing on the processed image through a weighted adaptive fusion encoder to generate depth feature data;

[0054] Step S40: The input feature data and deep feature data are fused by attention weights through a weighted adaptive fusion encoder to generate target feature tensor data;

[0055] Step S50: Process the target feature tensor data using a fully convolutional neural network to generate the target image.

[0056] In one embodiment, when performing step S10, the preprocessing steps may specifically include cropping, padding, and resizing. Here, the first frame image refers to the first frame image in the video sequence, which may contain the initial state of the target and its ground truth bounding box. The current image refers to the frame image in the video sequence that is being processed. The previous frame image refers to the frame image preceding the current image, used to provide the target's location.

[0057] In one embodiment, for the first frame, the target region can be cropped based on the target's ground truth bounding box. For the current image, the target region can be cropped based on the target position in the previous frame. This is because the target may move during the video sequence, and using the search result from the previous frame can provide a rough target position, allowing for more precise cropping. The sizes of the cropped images may not be the same, for example, they could be (192, 192, 3) and (384, 384, 3).

[0058] In one embodiment, to prevent the target from becoming too small during cropping and affecting the accuracy of subsequent processing, padding can be applied around the cropped image, such as adding solid color areas (e.g., white, black, or other background colors). This ensures that the input image contains sufficient information in the network. Specifically, if the size of the cropped image is smaller than a preset size, solid color padding will be applied to the missing parts of the image to bring it to the preset size requirement, maintaining image consistency and enabling the network to process fixed-size inputs.

[0059] In one embodiment, since deep learning models typically require input images of a fixed size, resizing can be used to ensure that the final size of the first initial image and the current initial image are the same as the preset size, thereby ensuring processing consistency and computational efficiency in the network.

[0060] In one embodiment, processing the first frame and the current image ensures the stability and robustness of target tracking, reducing the impact of accumulated errors and short-term variations. Although using the previous and current frames for tracking also has certain advantages, for the sake of overall efficiency and adaptability to specific tasks, this embodiment does not choose to use the previous and current frames for processing.

[0061] Specifically, the purpose of target tracking is to continuously track a specific target within a video sequence. To ensure tracking stability, issues such as target appearance changes and tracking drift need to be addressed. Target appearance changes refer to the various changes the target may undergo within the video sequence, such as deformation, lighting changes, and occlusion. These changes can cause significant differences in the target's appearance between different frames. Tracking drift refers to the phenomenon where relying solely on the previous and current frames for tracking can lead to accumulated errors, causing the tracked target to gradually deviate from its actual position.

[0062] Furthermore, the first frame image contains the target's initial state and the most accurate ground truth. The target's appearance information is most reliable and clearest in the first frame. The first frame image provides the target's baseline features, which is crucial for subsequent tracking. Using the first frame image as a reference point avoids misjudgments caused by short-term changes. Even if the target has undergone significant changes in the frames preceding the current frame, the features in the first frame image can still serve as a stable reference. By processing the first frame image and the current image, we can better address changes in the target's appearance and tracking drift. The first frame image provides a baseline for the initial state, while the current image provides the latest target information; combining these two improves tracking robustness.

[0063] Furthermore, the target position and feature information in the previous frame have already undergone one step of tracking processing, and may contain some errors. If tracking relies solely on the previous and current frames, these errors will gradually accumulate, leading to a decrease in tracking accuracy. The target in the previous frame may be affected by short-term changes (such as rapid movement or momentary occlusion), which may interfere with the tracking results in the current frame. The previous frame, as a reference point, is easily affected by the processing results of the previous frame, causing instability in the reference point throughout the tracking process and further increasing the risk of tracking drift.

[0064] Furthermore, in one embodiment, although using the previous and current frames for tracking can compensate for scale changes in the target in some cases, this method has relatively weak robustness and stability. Therefore, in this embodiment, the first frame and the current image are selected for processing to ensure the robustness of the tracking and its adaptability to specific tasks. In addition, there are some more complex multi-frame tracking methods, such as combining three or more frames, or even using extreme cases like seven twins. These methods can provide higher accuracy in specific scenarios, but they usually increase computational complexity and processing time.

[0065] In one embodiment, the acquired first frame initial image and the current initial image need to undergo two different network streams for processing. A network stream refers to a different processing path. The first network stream is the path that does not require deep fusion, and the second network stream is the path that requires deep fusion.

[0066] In one embodiment, when performing step S20, specifically, the first initial image and the current initial image can be processed through the first network stream. Both the first initial image and the current initial image are processed by two different Transformer encoders to generate connected features (features extracted by ViT (Vision Transformer)). That is, the first initial image and the current initial image are respectively input into two different encoders. The features generated by the two encoders are then concatenated to form a single feature, represented as the input feature data F.

[0067] In one embodiment, the encoder described above can be a weighted adaptive fusion encoder, which represents an encoder that fuses multimodal or multimodal information based on attention weights. Its purpose is to integrate multimodal or multimodal information using attention weights. Attention weights refer to weighting the importance of different information, thereby enabling the model to better focus on important parts and improve the information fusion effect. Multimodal information refers to data from different modalities (e.g., images from different cameras, or images from the same camera at different times).

[0068] In one embodiment, when performing step S30, step S30 may specifically include the following steps:

[0069] Step S31: Perform depth estimation processing on the first initial image and the current initial image respectively to generate the corresponding first frame depth image and current depth image;

[0070] Step S32: Normalize the first frame depth image and the current depth image according to the center depth value of the previous frame image to generate the corresponding first frame depth enhancement image and current depth enhancement image.

[0071] Step S33: Perform feature extraction processing on the first depth-enhanced image and the current depth-enhanced image to generate depth feature data.

[0072] In one embodiment, when executing step S31, specifically, the first initial image and the current initial image can be processed through a second network stream. First, the depth estimation processing of the first initial image and the current initial image can be performed separately using the MiDaS (Monocular Depth Awareness for Single Image) method to generate corresponding first-frame depth images and current depth images. The MiDaS method is a monocular depth estimation method capable of inferring scene depth information from a single image.

[0073] In one embodiment, when performing step S32, specifically, the center depth value of the previous frame image is used for normalization processing. The center depth value (Dc) of the previous frame image can be obtained from the depth image of the previous frame. The previous frame image refers to the frame preceding the current image; for example, if the current image is frame 141, the previous frame image refers to frame 140. The center depth value (Dc) of the previous frame image can be used to normalize the first frame depth image D. s (x,y) and the current depth image D t (x,y) is normalized to generate the corresponding first frame depth-enhanced image I′. s (x,y) and the current depth-enhanced image I′ t (x,y). I s (x,y) represents the pixel values ​​of the first frame's depth image. t (x,y) represents the pixel value of the current depth image. (x,y) represents the position in the image.

[0074] In one embodiment, the first frame depth-enhanced image I′ s (x,y) can be represented as: I′ s (x, y) = D′ s (x, y)⊙I s (x, y), ⊙ represents element-wise multiplication. The current depth-enhanced image can be represented as: I′ t (x, y) = D′ t (x, y)⊙I t (x, y), Specifically, for each pixel depth value in the first and current depth images, if the value is less than Dc, it is scaled down proportionally; if the value is greater than or equal to Dc, it is scaled up proportionally. This setting creates a depth contrast effect around the target, making the depth information of the target area more apparent.

[0075] In one embodiment, when step S33 is executed, specifically, the first depth-enhanced image and the current depth-enhanced image are processed by two different weighted adaptive fusion encoders to generate connected features (features extracted by ViT (Vision Transformer)). That is, the first depth-enhanced image and the current depth-enhanced image are respectively input into two different encoders. The features generated by the two encoders are then concatenated to form a whole feature, represented as depth feature data D.

[0076] In one embodiment, when performing step S40, step S40 may specifically include the following steps:

[0077] Step S41: The input feature data and the depth feature data are processed by the initial normalization layer of the weight adaptive fusion encoder to generate corresponding input normalized data and depth normalized data.

[0078] Step S42: The input normalized data and depth normalized data are processed by the linear layer of the weighted adaptive fusion encoder respectively, and the corresponding input attention parameters and depth attention parameters are calculated.

[0079] Step S43: The input attention parameters and the depth attention parameters are processed by the softmax layer of the weight adaptive fusion encoder to generate corresponding input attention weights and depth attention weights.

[0080] Step S44: Calculate the inner product of the input attention weights and the median vector matrix of the input attention parameters to generate input summary data; calculate the inner product of the deep attention weights and the median vector matrix of the deep attention parameters to generate deep summary data;

[0081] Step S45: The input attention weights and deep attention weights are processed through the attention weight fusion layer of the weight adaptive fusion encoder to generate corresponding hybrid attention weight values;

[0082] Step S46: The hybrid attention weight value, the input summary data, and the input feature data are processed by the feature-weight fusion layer of the weight adaptive fusion encoder to generate corresponding input merged feature data; the hybrid attention weight value, the depth summary data, and the depth feature data are processed by the feature-weight fusion layer to generate corresponding depth merged feature data.

[0083] Step S47: The input merged feature data and the depth merged feature data are processed by the output normalization layer of the weight adaptive fusion encoder, and the processing results are input into the fully connected layer of the weight adaptive fusion encoder for further processing to generate corresponding input feature tensor data and depth feature tensor data.

[0084] Step S48: The input feature tensor data and the depth feature tensor data are subjected to multiple attention weight fusion processes by the weight adaptive fusion encoder to generate target feature tensor data.

[0085] Referring to Figure 2, in one embodiment, when step S41 is performed, specifically, Layer Normalization (LN) refers to a normalization technique used in deep learning models to stabilize and accelerate the training process of the model by normalizing the input of each neural network layer.

[0086] Referring to Figure 2, in one embodiment, the input feature data F and the deep feature data D can be normalized by the initial normalization layer (LN) of the weighted adaptive fusion encoder to generate the corresponding input normalized data F. norm and depth normalized data D norm Among them, F norm =LayerNorm(F), D norm =LayerNorm(D). LayerNorm() represents the layer normalization operation, which is used to normalize the features of each sample. The initial normalization layer refers to the LN layer in Figure 2 that is close to the input feature data (Feature, F) and the depth feature data (D).

[0087] Referring to Figure 2, in one embodiment, when step S42 is executed, specifically, in the Transformer architecture, linear layers are typically used to map input data from one feature space to another. The output of a linear layer can be viewed as a linear transformation of the input data. In the Transformer, the attention mechanism is used to weight different parts of the input data, thereby focusing on important information. In this embodiment, the normalized input data F can be calculated separately using linear layers. norm and depth normalized data D norm Attention parameters include the query vector, key vector, and value vector, which determine the correlation and importance between different input parts.

[0088] Referring to Figure 2, in one embodiment, the input normalized data F norm The attention parameters can be represented as input attention parameters, which are denoted as Q. F ,K F V F Q F ,K F V F =Transform(F norm ). Deep normalized data D norm The attention parameters can be represented as the deep attention parameters Q. D ,K D V D Q D ,K D V D =Transform(D norm Here, Transform() represents the operation in the linear layer of the Transformer, used to calculate the attention parameters. The linear layer of the Transformer refers to Linear(Lin) in Figure 2.

[0089] Referring to Figure 2, in one embodiment, when performing step S43, specifically, for the calculation process of the input attention weights, the query vector Q in the input attention parameters can be calculated first. F transpose of the key vector The inner product of the matrices is then used, and subsequently, the softmax function can be applied to the inner product result to obtain the input attention weights A. F , Here, · denotes the inner product of matrices, and T represents the matrix transpose operation. The softmax function transforms the attention score matrix into a probability distribution matrix, where each element represents the attention weight of a certain input feature on other features. Similarly, the deep attention weights A can be calculated. D , Q D and K D These represent the query vector and key vector in the deep attention parameters, respectively. The softmax layer refers to the Softmax layer in Figure 2.

[0090] Referring to Figure 2, in one embodiment, when performing step S44, specifically, input summary data A′ is generated by calculating the inner product of the input attention weights and the median vector matrix of the input attention parameters. F , represented as: A′ F =A F ·V F , where V F This is represented as a value vector in the input attention parameters. The deep attention summary data A′ is generated by calculating the dot product of the deep attention weights and the value vector matrix of the deep attention parameters. D A′ D =A D ·V D , where V D This is represented as a value vector in the deep attention parameters. The input summary data and the deep summary data can be intermediate results, representing the weighted summaries of each feature after the attention mechanism is applied. For example, the input summary data reflects which parts of the input features are emphasized, and the deep summary data reflects which parts of the deep features are emphasized.

[0091] Referring to Figure 2, in one embodiment, when step S45 is executed, specifically, the input attention weight matrix and the deep attention weight matrix are concatenated through the attention weight fusion layer of the weighted adaptive fusion encoder to generate a hybrid attention weight matrix. The concatenation operation refers to connecting the input attention weight matrix and the deep attention weight matrix along a certain dimension (e.g., the feature dimension). By concatenating the input attention weights and the deep attention weights and calculating the adaptive hybrid attention weights, dynamic fusion of multimodal feature information can be achieved. Simultaneously, by directly calculating the sum of the input and deep attention weights, a simplified hybrid attention weight value can be generated for subsequent feature weighting operations. The hybrid attention weight value W... AMF Represented as: W AMF =A F +A DThe attention weight fusion layer refers to the Element-wise Additon(+) connected to the softmax layer in Figure 2.

[0092] Referring to Figure 2, in one embodiment, when step S46 is executed, specifically, the input summary data A′ is processed through the feature-weight fusion layer of the weight adaptive fusion encoder. F With the mixed attention weight value W AMF Perform element-wise multiplication and calculate the sum of the processed result and the input feature data F to generate the combined input feature data B. F B F =F+(A′) F ⊙W AMF The feature-weight fusion layer of the weighted adaptive fusion encoder is used to summarize the deep data A′. D With the mixed attention weight value W AMF Perform element-wise multiplication and calculate the sum of the result and the depth feature data D to generate depth-merged feature data B. D B D =D+(A′) D ⊙W AMF By performing element-wise multiplication, the mixed attention weights can apply different weights to each feature of the input summary data and the deep summary data, making the model focus more on important features and ignore unimportant ones. Adding the weighted summary data to the original feature data enhances the important parts of the original feature data while preserving the information in the original feature data. The feature-weight fusion layer refers to Element-wise Multiplieation (×) and Element-wise Addition (+) in Figure 2.

[0093] Referring to Figure 2, in one embodiment, when performing step S47, specifically, the input merged feature data B can be processed by the output normalization layer of the weighted adaptive fusion encoder. F Merging feature data with depth B D Layer normalization is performed, and the results are then fed into the fully connected layers of the weighted adaptive fusion encoder for further processing. This produces the output of the current layer of the encoder module, which is represented as the input feature tensor data F. feature and depth feature tensor data D depth_feature F feature =FC(LayerNorm(B F )), D depth_feature =FC(LayerNorm(B DIn this context, the output normalization layer refers to the LN layer near the feature-weight fusion layer in Figure 2, and the fully connected layer refers to the Fully Connected Layer (FC) in Figure 2.

[0094] In one embodiment, when performing step S48, specifically, the input feature tensor data F can be processed by a weighted adaptive fusion encoder. feature and depth feature tensor data D depth_feature Attention weight fusion is performed, and target feature tensor data is obtained from the final output input feature tensor data, and target depth tensor data is obtained from the final output depth feature tensor data. For example, the input feature tensor data and depth feature tensor data can be used as inputs to subsequent layers in a sequence of encoder modules that fuse multimodal or multimodal information based on attention weight fusion. After multiple (e.g., 12) encoder fusion processes based on attention weight fusion of multimodal or multimodal information, the final feature used is the one that does not go through the depth fusion path, i.e., the target feature tensor data is obtained from the final output input feature tensor data. Through the attention mechanism (multimodal fusion) of each layer, the correlation between input features and depth features can be dynamically learned, thereby fusing multimodal information more effectively. The normalization and transformation operations of each layer can enhance the expressive power of features, reduce the risk of overfitting, and improve the generalization ability of the model. Since the final output is the feature that does not go through the depth fusion path (i.e., the input feature tensor data), more original input feature information can be retained, and multimodal information can be used to assist and improve the accuracy of the model. Therefore, the final target depth tensor data can be ignored.

[0095] In one embodiment, when performing step S50, step S50 may specifically include the following steps:

[0096] Step S51: Process the target feature tensor data using a fully convolutional neural network to generate the corresponding target classification score map, normalization size parameter, and local offset matrix;

[0097] Step S52: Determine the initial position of the target by locating the peak value in the target classification score map;

[0098] Step S53: Correct the initial position based on the local offset matrix to determine the final position of the target;

[0099] Step S54: Scale the set bounding box according to the normalized size parameter, and encapsulate the final position with the scaled bounding box to generate the target image.

[0100] In one embodiment, when performing step S51, specifically, a fully convolutional neural network (FCN) is a network structure that replaces the fully connected layers of a traditional convolutional neural network (CNN) with convolutional layers, enabling it to process input images of arbitrary size and output feature maps of the same or similar size as the input image. The FCN can output a target classification score map, a normalized size parameter, and a local offset matrix. The target classification score map C is a two-dimensional matrix, where each element Cmn is in the range [0,1], representing the probability that position (m,n) is the target. The normalized size parameter Σ represents the scaling degree and is in the range [0,1]. The local offset matrix Δ represents the offset of the target position relative to the center. Here, Hs and Ws represent the height and width of the current initial image, respectively.

[0101] In one embodiment, when performing step S52, specifically, the initial position of the target can be determined by locating the peak value in the target classification score map. The initial position can be represented as (x... * ,y * ), (x * ,y * ) = argmax (m,n) Cmn, argmax represents the maximization operation.

[0102] In one embodiment, when executing steps S53 and S54, specifically, the local offset matrix is ​​a two-dimensional matrix with two values ​​at each location, representing the offset of the target center relative to the center of the grid cell. The initial position is corrected based on the local offset matrix to reduce spatial quantization errors. The bounding box is a pre-defined candidate box used for locating and scaling the target. The bounding box can be represented as (x, y, w, h), where x represents the horizontal coordinate of the upper-left corner of the bounding box, y represents the vertical coordinate of the upper-left corner of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box. Subsequently, the bounding box can be scaled using a normalized size parameter. Finally, the scaled bounding box can be used to encapsulate the final position to generate the target image (b). x ,b y ,b w ,b h ), represented as: (b x ,b y ,b w ,b h )=(x * +Δx,y * +Δy,Σw,Σh), where Δx and Δy refer to the offsets, and Σw and Σh refer to the deviations of the width and height of the target image relative to the scaled bounding box.

[0103] In one embodiment, during the model training phase, the composite loss function Ltotal comprehensively optimizes the model's classification and bounding box regression capabilities by combining the focus loss Lclass, L1 loss, and generalized IOU loss. Specifically, the L1 loss determines the center position and size of the regressed bounding box, ensuring that the predicted bounding box is as numerically close as possible to the ground truth bounding box. The generalized IOU loss optimizes the overlap of the bounding boxes while considering the size of the circumscribed rectangle, making the predicted bounding box not only accurate in position but also closer in shape to the ground truth bounding box.

[0104] As can be seen, the above scheme significantly improves the target tracking accuracy and robustness of robots in complex scenes by enhancing depth information, adaptively fusing multimodal information, and combining it with an attention mechanism. This method not only effectively addresses common problems such as dynamic scale changes and blurred targets, but also maintains high target recognition and tracking stability in complex environments. Its broad application prospects, especially in autonomous driving, industrial automation, and intelligent monitoring, provide important support for future technological development. Specifically, through enhanced depth information processing, the model can better understand the three-dimensional structure and spatial location of the target, thereby improving the accuracy of target tracking. Utilizing the attention mechanism, the model can dynamically focus on the salient features of the target, reducing background interference and false detections, further improving the accuracy of target tracking. Through the fusion of multimodal information, the model can utilize other sensor information (such as depth information) to assist in recognition and tracking even when the target image is blurred or partially occluded, improving robustness. The combination of depth information and the attention mechanism significantly enhances the model's environmental perception capability, enabling it to more accurately understand the surrounding environment and improve the stability of target tracking. In the field of autonomous driving, this method can provide more accurate and stable target tracking, helping vehicles better identify and track pedestrians, other vehicles, and other targets, improving driving safety. In industrial automation, this method can improve the accuracy of target detection and tracking for robots in dynamic environments, making it suitable for tasks such as object recognition and grasping on production lines. In the field of intelligent monitoring, this method can improve the accuracy of monitoring systems, enabling better identification and tracking of targets in complex scenarios and reducing false alarms and missed alarms.

[0105] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0106] Referring to Figure 3, the present invention also provides a target tracking processing apparatus, which can be applied to the above-described processing method. This processing apparatus may include a preprocessing module 100, a first extraction module 200, a second extraction module 300, a depth fusion module 400, and an image output module 500.

[0107] In one embodiment, the preprocessing module 100 can be used to preprocess the first frame image and the current image respectively to extract the target region in the image and generate the corresponding first frame initial image and current initial image.

[0108] In one embodiment, the first extraction module 200 can be used to perform feature extraction processing on the first initial image and the current initial image to generate input feature data.

[0109] In one embodiment, the second extraction module 300 can be used to perform deep fusion processing on the first initial image and the current initial image; and to perform feature extraction processing on the processed image to generate deep feature data.

[0110] In one embodiment, the deep fusion module 400 can be used to perform attention weight fusion processing on the input feature data and the deep feature data to generate target feature tensor data.

[0111] In one embodiment, the image output module 500 can be used to perform attention weight fusion processing on the input feature data and depth feature data to generate target feature tensor data.

[0112] Referring to Figure 4, in one embodiment, the electronic device 1 may include a memory 12, a processor 13, and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a target tracking program.

[0113] In one embodiment, the memory 12 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. Furthermore, the memory 12 can include both internal storage units and external storage devices of the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as target tracking processing, but also to temporarily store data that has been output or will be output.

[0114] In one embodiment, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 through various interfaces and lines. It executes programs or modules stored in the memory 12 and calls data stored in the memory 12 to perform various functions and process data of the electronic device 1.

[0115] In one embodiment, the processor 13 executes the operation module of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the target tracking processing method described above.

[0116] In one embodiment, a computer program may be divided into one or more modules, one or more of which are stored in memory 12 and executed by processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in electronic device 1.

[0117] The embodiments of the present invention disclosed above are merely illustrative of the invention. The embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A target tracking processing method, characterized in that, include: The first frame image and the current image are preprocessed separately to extract the target region in the image and generate the corresponding first frame initial image and current initial image; The first initial image and the current initial image are processed by a weighted adaptive fusion encoder to extract features and generate input feature data. The first initial image and the current initial image are subjected to deep fusion processing; and the processed image is subjected to feature extraction processing by the weighted adaptive fusion encoder to generate deep feature data. The input feature data and the deep feature data are fused by the weighted adaptive fusion encoder to generate target feature tensor data. The target image is generated by processing the target feature tensor data using a fully convolutional neural network.

2. The target tracking processing method according to claim 1, characterized in that, The step of performing deep fusion processing on the first initial image and the current initial image includes: Depth estimation processing is performed on the first initial image and the current initial image respectively to generate the corresponding first-frame depth image and current depth image; Based on the center depth value of the previous frame image, the first frame depth image and the current depth image are normalized respectively to generate the corresponding first frame depth-enhanced image and current depth-enhanced image.

3. The target tracking processing method according to claim 1, characterized in that, The step of performing attention weight fusion processing on the input feature data and the deep feature data through the weight adaptive fusion encoder to generate target feature tensor data includes: The input feature data and the depth feature data are processed by the initial normalization layer of the weight adaptive fusion encoder to generate corresponding input normalized data and depth normalized data. The linear layer of the weighted adaptive fusion encoder processes the input normalized data and the depth normalized data respectively, and calculates the corresponding input attention parameters and depth attention parameters. The input attention parameters and the depth attention parameters are processed by the softmax layer of the weight adaptive fusion encoder to generate corresponding input attention weights and depth attention weights. Calculate the inner product of the input attention weights and the median vector matrix of the input attention parameters to generate input summary data; calculate the inner product of the deep attention weights and the median vector matrix of the deep attention parameters to generate deep summary data; The input attention weights and deep attention weights are processed by the attention weight fusion layer of the weight adaptive fusion encoder to generate corresponding hybrid attention weight values; The feature-weight fusion layer of the weight adaptive fusion encoder processes the hybrid attention weight values, the input summary data, and the input feature data to generate corresponding input merged feature data; the feature-weight fusion layer processes the hybrid attention weight values, the depth summary data, and the depth feature data to generate corresponding depth merged feature data. The input merged feature data and the depth merged feature data are processed by the output normalization layer of the weight adaptive fusion encoder, and the processing results are input into the fully connected layer of the weight adaptive fusion encoder for further processing to generate corresponding input feature tensor data and depth feature tensor data. The input feature tensor data and the depth feature tensor data are subjected to multiple attention weight fusion processes by the weight adaptive fusion encoder to generate the target feature tensor data.

4. The target tracking processing method according to claim 3, characterized in that, The hybrid attention weight value W AMF , represented as: W AMF =A F +A D , where A F A represents the input attention weights. D This represents the weights for deep attention.

5. The target tracking processing method according to claim 4, characterized in that, The input merged feature data B F , represented as: B F =F+(A′) F ⊙W AMF ), where F represents the input feature data, A' F This represents the input summary data, and ⊙ represents element-wise multiplication; the deep merging feature data B D , represented as: B D =D+(A′) D ⊙W AMF ), where D represents depth feature data, A' D This is represented as deeply summarized data.

6. The target tracking processing method according to claim 3, characterized in that, The step of generating target feature tensor data by performing multiple attention weight fusion processes on the input feature tensor data and the depth feature tensor data through the weight adaptive fusion encoder includes: The input feature tensor data and the depth feature tensor data are subjected to multiple attention weight fusion processes by the weight adaptive fusion encoder, and the target feature tensor data is obtained based on the final output input feature tensor data.

7. The target tracking processing method according to claim 1, characterized in that, The step of processing the target feature tensor data using a fully convolutional neural network to generate the target image includes: The target feature tensor data is processed by a fully convolutional neural network to generate the corresponding target classification score map, normalization size parameter, and local offset matrix. The initial position of the target is determined by locating the peak value in the target classification score map; The initial position is corrected based on the local offset matrix to determine the final position of the target; The set bounding box is scaled according to the normalized size parameter, and the final position is encapsulated by the scaled bounding box to generate the target image.

8. A target tracking processing device, characterized in that, include: The preprocessing module is used to preprocess the first frame image and the current image respectively to extract the target region in the image and generate the corresponding first frame initial image and current initial image; The first extraction module is used to perform feature extraction processing on the first frame initial image and the current initial image to generate input feature data; The second extraction module is used to perform depth fusion processing on the first frame initial image and the current initial image; and to perform feature extraction processing on the processed image to generate depth feature data; The deep fusion module is used to perform attention weight fusion processing on the input feature data and the deep feature data to generate target feature tensor data; as well as The image output module is used to process the target feature tensor data to generate the target image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes a computer program to implement the steps of the target tracking processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the steps of the target tracking processing method as described in any one of claims 1 to 7.

Images