Data processing method, apparatus and device, vehicle, medium, and program product

Abstract

The present disclosure provides a data processing method, apparatus and device, a vehicle, a medium, and a program product. The method comprises: acquiring millimeter-wave radar point voxel data corresponding to millimeter-wave radar data; encoding the millimeter-wave radar point voxel data to generate a target voxel sequence feature; performing multi-layer linear computation on the target voxel sequence feature to generate a query sequence feature, a key sequence feature, and a value sequence feature; computing the query sequence feature and the key sequence feature on the basis of an attention mechanism to obtain an autocorrelation matrix; and encoding the autocorrelation matrix and the value sequence feature, and performing residual connection on an encoding result and the target voxel sequence feature to generate a millimeter-wave radar autocorrelation-enhanced sequence feature.

Description

A data processing method, apparatus, equipment, vehicle, medium, and program product Cross-references to related applications

[0001] This application claims priority to Chinese patent application No. 202411544643.2, filed on October 31, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of target detection technology, specifically to a data processing method, apparatus, equipment, vehicle, medium, and program product. Background Technology

[0003] Currently, in order to improve driving safety and convenience as well as enhance the performance of intelligent driving systems, vehicles are often equipped with a variety of sensors such as LiDAR, millimeter-wave radar, and cameras. By fusing data from different types of sensors, more comprehensive and accurate information about the vehicle's surrounding environment can be obtained, providing a data foundation for target detection. This enables better support for assisted driving functions such as automatic following, adaptive cruise control, lane keeping, and automatic parking.

[0004] Millimeter-wave radar data often suffers from sparsity due to limitations in resolution, environmental interference, multipath effects, and the shape, size, material, and surface roughness of the target. However, related technologies for multi-sensor data fusion typically ignore this sparsity, leading to mismatches and consequently poor target detection accuracy. Therefore, improving the sparsity of millimeter-wave radar data to enhance its utilization efficiency is a pressing issue that needs to be addressed. Summary of the Invention

[0005] This disclosure provides a data processing method, apparatus, equipment, vehicle, medium, and program product that can effectively improve the sparsity problem of Radar data and enhance the utilization efficiency of Radar data.

[0006] A first aspect of this disclosure provides a data processing method, comprising: acquiring millimeter-wave radar point voxel data corresponding to millimeter-wave radar data; encoding the millimeter-wave radar point voxel data to generate target voxel sequence features; performing multi-level linear calculations on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features; calculating the query sequence features and the key sequence features based on an attention mechanism to obtain an autocorrelation matrix; and encoding the autocorrelation matrix and the value sequence features, and performing residual concatenation of the encoding result and the target voxel sequence features to generate autocorrelation-enhanced sequence features of the millimeter-wave radar.

[0007] A second aspect of this disclosure provides a data processing apparatus, including an autocorrelation point enhancement module, configured to: acquire millimeter-wave radar point voxel data corresponding to millimeter-wave radar data; encode the millimeter-wave radar point voxel data to generate target voxel sequence features; perform multi-level linear calculations on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features; calculate the query sequence features and the key sequence features based on an attention mechanism to obtain an autocorrelation matrix; encode the autocorrelation matrix and the value sequence features, and perform residual concatenation of the encoding result and the target voxel sequence features to generate autocorrelation-enhanced sequence features of the millimeter-wave radar.

[0008] A third aspect of this disclosure provides a data processing apparatus, the data processing apparatus including a processor, a memory, and a data processing program stored in the memory and executable by the processor, wherein when the data processing program is executed by the processor, it implements the steps of the data processing method as described above.

[0009] A fourth aspect of this disclosure provides a vehicle including a vehicle body and the data processing device described in the third aspect above.

[0010] A fifth aspect of this disclosure provides a computer-readable storage medium storing a data processing program, wherein when executed by a processor, the data processing program implements the steps of the data processing method described above.

[0011] A sixth aspect of this disclosure provides a computer program product including computer instructions that, when executed by a processor, implement the steps of the data processing method provided in the first aspect. Attached Figure Description

[0012] Figure 1 is a schematic flowchart of a data processing method according to some embodiments of the present disclosure;

[0013] Figure 2 is a schematic diagram of Pillar encoding methods in some embodiments of this disclosure;

[0014] Figure 3 is a schematic diagram of feature allocation in some embodiments of this disclosure;

[0015] Figure 4 is a schematic diagram of the target detection process according to some embodiments of this disclosure;

[0016] Figure 5 is a schematic diagram of the feature fusion process of some embodiments of this disclosure;

[0017] Figure 6 is a schematic diagram of the hardware structure of a data processing device according to some embodiments of the present disclosure. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present disclosure.

[0019] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0020] In a first aspect, embodiments of this disclosure provide a data processing method.

[0021] Figure 1 is a flowchart illustrating an embodiment of the data processing method. As shown in Figure 1, the data processing method includes:

[0022] Step S10: Obtain millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data;

[0023] Step S20: Encode the millimeter-wave radar point voxel data to generate target voxel sequence features;

[0024] Step S30: Perform multi-level linear computation on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features;

[0025] Step S40: Calculate the autocorrelation matrix by analyzing the query sequence features and key sequence features based on the attention mechanism;

[0026] Step S50: Encode the autocorrelation matrix and value sequence features through a multi-head self-attention mechanism, and perform residual connection between the encoding results and the target voxel sequence features to generate autocorrelation enhancement sequence features of millimeter-wave radar.

[0027] In some embodiments, obtaining the millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data may include: obtaining the total number of collected millimeter-wave radar points and the information contained in each millimeter-wave radar point; generating a point set with a total number and an information dimension based on the total number of millimeter-wave radar points and the information contained in each millimeter-wave radar point; and performing voxelization processing on the point set to obtain the millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data.

[0028] In one implementation, the mathematical expression for millimeter-wave radar points is first defined as follows: In the formula, N represents the total number of millimeter-wave radar points collected by the millimeter-wave radar in a certain driving scenario; D represents the information contained in a single millimeter-wave radar point, such as the coordinates (x, y, z) of the millimeter-wave radar point, the reflection value of the millimeter-wave radar point (the reflection value corresponds to the confidence level of the millimeter-wave radar point; it should be noted that if the reflection value is below a certain threshold, the millimeter-wave radar point will not be recorded), and the speed value of the millimeter-wave radar point, etc.; P Radar Let P represent an N×D dimensional point set. Then, for P... Radar Voxelization is performed, during which the three-dimensional space is divided into a series of small cubes (i.e., voxels) according to a preset voxel size, and P... Radar Each point in the data is assigned to a corresponding voxel to obtain millimeter-wave radar point voxel data. It should be noted that the specific methods and principles of voxelization are common knowledge in this field, so for the sake of brevity, they will not be elaborated here.

[0029] In step S20: the millimeter-wave radar point voxel data is encoded to generate target voxel sequence features.

[0030] It should be understood that currently, after completing the voxelization of millimeter-wave radar points, the millimeter-wave radar point voxel data is usually directly encoded into BEV (Bird's Eye View) features, thus neglecting the exploration of the point-to-point relationships within the millimeter-wave radar, resulting in the sparseness of the generated BEV features. Therefore, to solve the data sparsity problem, this embodiment uses Pillar encoding (a coding method that divides 3D point cloud data into a series of uniformly distributed 2D grids Pillars on the xy plane) to encode the voxelized data (i.e., millimeter-wave radar point voxel data) into a set of voxel features, thus obtaining the target voxel sequence features, which can be represented mathematically as follows: Where N is the same as P mentioned above. Radar In the diagram, N corresponds to m, which represents P. Radar The D-dimensional information in the table is encoded as a dimension, the specific value of which can be determined according to actual needs. For example, if m = 16, then table P... RadarThe D-dimensional information is encoded as 16 dimensions.

[0031] Figure 2 is a schematic diagram of the Pillar encoding method. It should be noted that Pillar encoding is used to encode millimeter-wave radar point voxel data into target voxel sequence features f. radar The main approach is to uniformly encode all voxel regions at each entire height after voxel division (i.e., all millimeter-wave radar points contained in the dashed box in Figure 2), rather than encoding each voxel block region, in order to avoid generating invalid features. However, the specific encoding method and principle are common knowledge in this field, so for the sake of simplicity, they will not be elaborated here.

[0032] In step S30: Perform multi-level linear computation on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features.

[0033] It is understandable that after encoding using the aforementioned Pillar encoding method, P can be... Radar Encoding is performed as sequential features to transform the information of millimeter-wave radar points from their specific geometric locations to a high-dimensional space. To improve the sparsity of millimeter-wave radar in this high-dimensional space, this embodiment proposes an autocorrelation point enhancement method based on attention mechanisms (such as the Transformer attention mechanism). This method enhances the point-to-point relationships among millimeter-wave radar points using an autocorrelation point enhancement module, resulting in improved millimeter-wave radar point features.

[0034] In some embodiments, a multilevel linear computation unit (MLP) is used to process the target voxel sequence features f. radar High-dimensional encoding is performed to obtain query sequence features Q, key sequence features K, and value sequence features V. In one implementation, the target voxel sequence features are calculated based on MLP (16, 32, 64) as follows, so that the three sets of parameters Q, K, and V are encoded into 64 dimensions, i.e.

[0035]

[0036]

[0037]

[0038] In the formula, W Q W K W V They represent different neural network learning parameters, and W K =W V It should be noted that W Q W K W VThe specific value can be determined according to actual needs, and is not limited here.

[0039] In step S40: The autocorrelation matrix is ​​obtained by calculating the query sequence features and key sequence features based on the attention mechanism.

[0040] In some embodiments, calculating the autocorrelation matrix based on the query sequence features and key sequence features using an attention mechanism may include: obtaining the autocorrelation matrix based on an activation function, the query sequence features, the transpose of the key sequence features, and the feature channel dimension of the key sequence features. The formula for calculating the autocorrelation matrix may be:

[0041]

[0042] In the formula, M auto Let K represent the autocorrelation matrix, Q represent the query sequence features, and K represent the autocorrelation matrix. T C represents the transpose of the key sequence features. K The feature channel dimension represents the key sequence features.

[0043] In one implementation, to obtain the autocorrelation-based millimeter-wave radar point enhancement results, the correlation between the query sequence feature Q and the key sequence feature K is calculated using an attention mechanism; wherein, since the parameters of both the query sequence feature Q and the key sequence feature K are derived from the target voxel sequence feature f radar Therefore, the autocorrelation matrix can be obtained through the attention mechanism. The specific formula for calculating the autocorrelation matrix is ​​as follows:

[0044]

[0045] Here, SelfAttention represents Softmax computation, meaning it uses the Softmax activation function to calculate the correlation between the query sequence feature Q and the key sequence feature K:

[0046]

[0047] In the formula, C K K represents the feature channel dimension of the key sequence features. T This represents the transpose of the key sequence features.

[0048] Therefore, based on equations (2) and (3), the formula for calculating the autocorrelation matrix can be obtained as follows:

[0049] In step S50: The autocorrelation matrix and value sequence features are encoded through a multi-head self-attention mechanism, and the encoding results are residually connected with the target voxel sequence features to generate millimeter-wave radar autocorrelation enhancement sequence features.

[0050] In some embodiments, encoding the autocorrelation matrix and value sequence features may include encoding the autocorrelation matrix and value sequence features using a multi-head self-attention mechanism. The expression for the millimeter-wave radar autocorrelation enhancement sequence features is as follows:

[0051]

[0052] In the formula, This represents the autocorrelation enhancement sequence features of millimeter-wave radar, where MultiHeadAttention represents the multi-head attention module, and M... auto Let V represent the autocorrelation matrix, V represent the value sequence features, head represent the number of heads in the multi-head attention module, and f represent the autocorrelation matrix. radar This indicates the characteristics of the target voxel sequence.

[0053] To ensure the inherent consistency of the original millimeter-wave radar point sequence features, a multi-head self-attention encoding method will be used to encode the autocorrelation matrix Meauto and the value sequence features V. The encoding results will then be combined with the target voxel sequence features f. radar Residual joins are performed to obtain enhanced autocorrelation features of millimeter-wave radar, i.e., to generate enhanced autocorrelation sequence features of millimeter-wave radar. This effectively improves the sparsity of millimeter-wave radar data, thereby enhancing its utilization efficiency. The calculation formula for the enhanced autocorrelation sequence features of millimeter-wave radar is as follows:

[0054]

[0055] In the formula, Representing the autocorrelation enhancement sequence features of millimeter-wave radar, MultiHeadAttention represents a multi-head self-attention module, which requires three input parameters: the autocorrelation matrix M. auto The value sequence features V and the number of heads corresponding to the multi-head attention module are also considered. It is worth noting that this embodiment uses residual connections to superimpose the original sequence features, which effectively stabilizes the network training and ensures network convergence. Understandably, considering the sparsity of millimeter-wave radar data, this embodiment implements an autocorrelation point enhancement method through the aforementioned steps to improve the sensing capability of millimeter-wave radar features. Furthermore, the data enhancement operation can be performed with lower computational load, thus ensuring network efficiency.

[0056] In other embodiments, encoding the autocorrelation matrix and value sequence features may include encoding the autocorrelation matrix and value sequence features through methods such as correlation calculation or neural network encoding.

[0057] As can be seen, this embodiment generates target voxel sequence features by encoding the voxel data of millimeter-wave radar points corresponding to the millimeter-wave radar data, thereby transforming the millimeter-wave radar point information from specific geometric locations to a high-dimensional space. Then, multi-level linear calculations are performed on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features, achieving high-dimensional encoding. Next, to obtain autocorrelation-enhanced millimeter-wave radar points, the correlation between query sequence features and key sequence features is calculated based on an attention mechanism to generate an autocorrelation matrix. Finally, to ensure the inherent connection of the original millimeter-wave radar point sequence features, multi-head self-attention encoding is used to encode the autocorrelation matrix and value sequence features. The encoded results and target voxel sequence features are then residually concatenated to generate millimeter-wave radar autocorrelation-enhanced sequence features. This effectively improves the sparsity of millimeter-wave radar data, enhances the utilization efficiency of millimeter-wave radar data, and thus improves the efficiency and accuracy of data fusion.

[0058] In some embodiments, after the step of generating the autocorrelation enhancement sequence features of the millimeter-wave radar, steps S100 to S101 are further included:

[0059] Step S100: Determine the index coordinates based on the voxel coordinates corresponding to the autocorrelation enhancement sequence features of the millimeter-wave radar and the sensing range of the bird's-eye view of the millimeter-wave radar;

[0060] Step S101: Based on the index coordinates, map the autocorrelation enhancement sequence features of the millimeter-wave radar to the bird's-eye view features of the millimeter-wave radar.

[0061] In this embodiment, although the autocorrelation enhancement sequence features of the millimeter-wave radar can be obtained after enhancing the intra-point correlation features of the millimeter-wave radar, However these features Since the features are discrete, this embodiment also includes an autocorrelation millimeter-wave radar feature allocation module to obtain the final feature distribution in the BEV view. By assigning features to the final BEV view space, the final millimeter-wave radar BEV feature can be obtained. R And BEV R The dimensions are H×W×C, where H represents BEV. R The length of W represents the BEV. R The width, C represents BEV R The number of channels; where H×W is the same as the BEV space size, while C is the same as... Same dimensions, for example Therefore, C is 64.

[0062] In some embodiments, the step of determining the sensing range of the bird's-eye view may include: acquiring the sensing range and voxel length of the millimeter-wave radar point cloud; determining the sensing grid size of the bird's-eye view based on the voxel length; and determining the sensing range of the bird's-eye view by comparing the sensing range of the point cloud with the sensing grid size of the bird's-eye view.

[0063] It should be noted that the size of the BEV space can be defined according to the commonly used autonomous driving perception range definition of millimeter-wave radar in driving scenarios. For example, assuming the perception range of the driving scenario is ±51.2m in front and back (y-axis), ±51.2m in left and right (x-axis), and a height range (z-axis) of -5m to +3m, since this embodiment only cares about the front and back and left and right perception ranges, the perception range of the point cloud can be determined based on the perception ranges corresponding to the front and back axes and the left and right axes. (Right now The BEV sensing mesh size is defined based on the voxel length h (e.g., if h is 0.8m, then the BEV sensing mesh size is also 0.8m), and then based on the sensing range of the point cloud... The sensing range of the BEV view can be calculated by combining the BEV sensing grid size with the BEV sensing grid size. Therefore, the size of the BEV space can be defined as 128×128, so H and W are both 128.

[0064] In this embodiment, the autocorrelation millimeter-wave radar feature allocation module determines the index coordinates of the autocorrelation enhancement sequence feature based on the voxel coordinates corresponding to the voxel containing the feature and the sensing range of the BEV (Body Activated Vehicle). This involves constructing a transformation relationship between voxel coordinates and BEV sensing grid coordinates to determine the index coordinates of the autocorrelation enhancement sequence feature. Then, a mapping relationship between the autocorrelation enhancement sequence feature and the BEV sensing grid is constructed using these index coordinates. Based on the index coordinates, the autocorrelation enhancement sequence feature can be allocated (e.g., cuboid S1 in the feature allocation diagram shown in Figure 3) to the corresponding BEV sensing grid (e.g., A1 in Figure 3), thus obtaining the millimeter-wave radar BEV feature BEVR. It should be noted that the voxel coordinates corresponding to each voxel are predefined; during feature allocation, the voxel coordinates are directly obtained for index coordinate calculation.

[0065] In some embodiments, the formula for calculating the index coordinates is:

[0066] coords = r·vy + vx

[0067] In the formula, coords represents the index coordinates, and r represents the perception range of the bird's-eye view, i.e. vy represents the vertical axis coordinate of the voxel, and vx represents the horizontal axis coordinate of the voxel.

[0068] In this embodiment, the voxel coordinates after the millimeter-wave radar voxel are first defined as (vz, vy, vx), and the voxel size of each millimeter-wave radar is defined (for example, the voxel size is: voxel height d = 3 - (-5) = 8m, voxel length h = 0.8m, voxel width w = 0.8m, which is consistent with the BEV sensing grid size). It can be understood that each millimeter-wave radar point will be assigned to a voxel. Therefore, the millimeter-wave radar autocorrelation enhancement sequence features obtained by encoding a voxel are... The index coordinates can be calculated from the voxel coordinates vy, vx and the sensing range r of BEV of the voxel, i.e., the autocorrelation enhancement sequence features of each millimeter-wave radar. The index coordinates can be calculated using the following formula:

[0069] coords = r·vy + vx

[0070] In the formula, coords represents the index coordinates; in this embodiment, after assigning the autocorrelation enhancement sequence features of the millimeter-wave radar to the corresponding BEV sensing grid based on the index coordinates, the millimeter-wave radar BEV feature BEV can be obtained through 2D convolutional coding. R .

[0071] In this embodiment, after autocorrelation millimeter-wave radar point feature allocation, the autocorrelation enhancement sequence features of the millimeter-wave radar can be... Assigned to the corresponding BEV view space, that is, through coordinate mapping. Transition to millimeter-wave radar BEV characteristics BEV R This is used for subsequent feature interaction. It is evident that this embodiment improves the accuracy and range of perception through precise feature allocation and the definition of the BEV view space, enabling the system to more accurately identify and locate targets.

[0072] In some embodiments, after the step of obtaining the millimeter-wave radar bird's-eye view features, steps S200 to S204 are further included:

[0073] S200: Acquire the LiDAR bird's-eye view features corresponding to the LiDAR data and the camera bird's-eye view features corresponding to the camera image data;

[0074] S201: Perform feature dimension transformation on the bird's-eye view features of millimeter-wave radar, lidar, and camera respectively, and stitch together all the transformed bird's-eye view features to obtain stitched features;

[0075] S202: Perform feature activation operations on the spliced ​​features using different activation functions to obtain different learnable attention weights;

[0076] S203: The learnable attention weights and their corresponding bird's-eye view features are weighted separately to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features, and the target camera bird's-eye view features.

[0077] S204: The bird's-eye view features of the target millimeter-wave radar, the target lidar, and the target camera are stitched together to generate fused features.

[0078] In this embodiment, adaptive fusion of three modalities—LiDAR, millimeter-wave radar, and camera data—will be achieved. Specifically, referring to the target detection process diagram shown in Figure 4, the millimeter-wave radar BEV feature (BEV) is first acquired. R LiDAR BEV characteristics corresponding to LiDAR data L And the camera BEV feature corresponding to the camera image data. I Among these methods, the LiDAR data can be sequentially processed through voxelization and 3D sparse convolutional encoding to obtain the LiDAR BEV feature (BEV). L Simultaneously, shallow feature extraction, deep feature extraction, image-view transformation based on LSS (Lift-Splat-Shoot, a bottom-up BEV perception method), and 2D convolutional encoding are performed on the camera image data to obtain the camera BEV feature (BEV). R It should be noted that the LiDAR BEV feature is generated by using LiDAR data and camera image data respectively. L BEV characteristics and camera I The methods and principles are common knowledge in this field, so for the sake of brevity, they will not be elaborated here.

[0079] Secondly, for BEV features from the three modalities, this embodiment designs a simple and efficient gating fusion module (i.e., the adaptive module AU in the feature fusion flowchart shown in Figure 5) to generate unified BEV features, enabling the model to adaptively fuse complementary modalities. Referring to Figure 5, BEV features are respectively... R BEV L and BEV I The feature dimension is transformed, and all the transformed BEV features are concatenated to obtain concatenated features. Then, different sigmoid activation functions are applied to the concatenated features to obtain three different sets of learnable attention weights (e.g., W). R WL and W I Each set of learnable attention weights corresponds to a transformed bird's-eye view feature. It should be noted that the type and parameter settings of the sigmoid function can be determined according to actual needs and are not limited here. Then, the three sets of learnable attention weights are used to weight the corresponding BEV features, i.e., attention weighting is used to weight W. R With BEV R W L With BEV L and W I With BEV I By multiplying them separately, a new target millimeter-wave radar BEV feature, BEV′, is obtained. R Target lidar BEV characteristics BEV′ L BEV characteristics of the target camera (BEV′) I Then for BEV′ R BEV′ L and BEV′ I By splicing the data, a fused feature can be generated.

[0080] Finally, the fused features are sent to the detection head module to complete the final 3D object detection task. It should be understood that the detection head module can use methods such as CenterPoint to achieve target detection. The CenterPoint algorithm refers to a method for target detection and tracking based on the center point. It first converts 3D point cloud data into a top-down feature map through a 3D encoder, and then uses the detection head to find the center point of the target and regress the target's 3D size, orientation, velocity and other attributes. Its specific implementation method and principle are common knowledge in this field, and will not be limited here for the sake of simplicity.

[0081] In some embodiments, the feature dimension transformation of the millimeter-wave radar bird's-eye view features, lidar bird's-eye view features, and camera bird's-eye view features includes:

[0082] The features of the millimeter-wave radar bird's-eye view are sequentially convolved, batch normalized, and activated to obtain the transformed millimeter-wave radar bird's-eye view features.

[0083] The LiDAR bird's-eye view features are sequentially subjected to convolution, batch normalization and activation operations to obtain the transformed LiDAR bird's-eye view features.

[0084] The camera bird's-eye view features are sequentially subjected to convolution, batch normalization, and activation operations to obtain the transformed camera bird's-eye view features.

[0085] As shown in Figure 5, the adaptive module AU is calculated using three different modal BEV features as input, and performs feature dimension transformation on the three different modal BEV features. That is, after the three different modal BEV features are processed by convolution, batch normalization and activation operation (i.e., CBR represents the commonly used Convolution + Batch Normalization + ReLU activation function operation), the transformed millimeter-wave radar BEV features, the transformed lidar BEV features and the transformed camera BEV features are obtained respectively.

[0086] This embodiment optimizes the sparsity of millimeter-wave radar data through autocorrelation point enhancement technology. By utilizing point-to-point relationships in the scene and learning the point-to-point autocorrelation mechanism within millimeter-wave radar points, it significantly improves the feature representation capability of radar data, thereby significantly improving the sparsity of millimeter-wave radar BEV features obtained by traditional methods, reducing reliance on pre-selected boxes, and thus enhancing the accuracy of target detection and the overall robustness of the system. Secondly, this embodiment effectively maps the enhanced radar point features to the BEV view space through an autocorrelation millimeter-wave radar point feature allocation module, ensuring the accurate spatial distribution of features and improving the positioning accuracy and adaptability of the perception system to complex environments. Therefore, adaptive fusion and autocorrelation enhancement can improve the model's adaptability to environmental changes, thereby enhancing the model's robustness. In addition, this embodiment adopts a full-modal adaptive fusion strategy, integrating BEV features from different sensors (such as cameras, LiDAR, and millimeter-wave radar). Through the designed adaptive module, the feature fusion process is optimized, achieving efficient integration of multimodal data to improve detection accuracy and robustness, thereby enhancing the overall performance and real-time performance of the detection system.

[0087] Secondly, embodiments of this disclosure also provide a data processing apparatus.

[0088] In one embodiment, the data processing apparatus includes an autocorrelation point enhancement module, which is used to:

[0089] Acquire millimeter-wave radar point voxel data corresponding to millimeter-wave radar data;

[0090] The millimeter-wave radar point voxel data is encoded to generate target voxel sequence features;

[0091] Multi-level linear computation is performed on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features;

[0092] The autocorrelation matrix is ​​obtained by calculating the query sequence features and key sequence features based on the attention mechanism;

[0093] The autocorrelation matrix and value sequence features are encoded, and the encoding results are residually concatenated with the target voxel sequence features to generate autocorrelation enhancement sequence features for millimeter-wave radar.

[0094] In some embodiments, the apparatus further includes an autocorrelation millimeter-wave radar feature allocation module, which is used for:

[0095] The index coordinates are determined based on the voxel coordinates corresponding to the autocorrelation enhancement sequence features of the millimeter-wave radar and the sensing range of the bird's-eye view.

[0096] Based on the index coordinates, the autocorrelation enhancement sequence features of the millimeter-wave radar are mapped into millimeter-wave radar bird's-eye view features.

[0097] In some embodiments, the formula for calculating the index coordinates is:

[0098] coords = r·vy + vx

[0099] In the formula, coords represents the index coordinates, r represents the perception range of the bird's-eye view, vy represents the vertical axis coordinate of the voxel, and vx represents the horizontal axis coordinate of the voxel.

[0100] In some embodiments, the device further includes an adaptation module for:

[0101] Acquire the LiDAR bird's-eye view features corresponding to the LiDAR data and the camera bird's-eye view features corresponding to the camera image data;

[0102] The feature dimensions of the bird's-eye view features from millimeter-wave radar, lidar, and camera are transformed respectively, and all the transformed bird's-eye view features are stitched together to obtain the stitched features.

[0103] Different activation functions are used to perform feature activation operations on the spliced ​​features to obtain different learnable attention weights;

[0104] The learnable attention weights and their corresponding bird's-eye view features are weighted separately to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features, and the target camera bird's-eye view features. The target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features, and the target camera bird's-eye view features are then stitched together to generate fused features.

[0105] In some embodiments, the adaptive module is specifically used for:

[0106] The features of the millimeter-wave radar bird's-eye view are sequentially convolved, batch normalized, and activated to obtain the transformed millimeter-wave radar bird's-eye view features.

[0107] The LiDAR bird's-eye view features are sequentially subjected to convolution, batch normalization and activation operations to obtain the transformed LiDAR bird's-eye view features.

[0108] The camera bird's-eye view features are sequentially subjected to convolution, batch normalization, and activation operations to obtain the transformed camera bird's-eye view features.

[0109] In some embodiments, the autocorrelation point enhancement module is specifically used to: encode the autocorrelation matrix and the value sequence features through a multi-head self-attention mechanism.

[0110] In some embodiments, the autocorrelation point enhancement module is specifically used to: obtain the total number of millimeter-wave radar points collected and the information contained in each millimeter-wave radar point; generate a point set with a total number and information dimension based on the total number of millimeter-wave radar points and the information contained in each millimeter-wave radar point; and perform voxelization processing on the point set to obtain millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data.

[0111] In some embodiments, the formula for calculating the autocorrelation matrix is:

[0112]

[0113] In the formula, M auto Let K represent the autocorrelation matrix, Q represent the query sequence features, and K represent the autocorrelation matrix. T C represents the transpose of the key sequence features. K The feature channel dimension represents the key sequence features.

[0114] In some embodiments, the expression for the autocorrelation enhancement sequence features of the millimeter-wave radar is:

[0115]

[0116] In the formula, This represents the autocorrelation enhancement sequence features of millimeter-wave radar, where MultiHeadAttention represents the multi-head attention module, and M... auto Let V represent the autocorrelation matrix, V represent the value sequence features, head represent the number of heads in the multi-head attention module, and f represent the autocorrelation matrix. radar This indicates the characteristics of the target voxel sequence.

[0117] The functions of each module in the above data processing device correspond to the steps in the above data processing method embodiments, and their functions and implementation processes will not be described in detail here.

[0118] Thirdly, embodiments of this disclosure provide a data processing device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0119] Referring to Figure 6, which is a schematic diagram of the hardware structure of the data processing device involved in the embodiments of this disclosure, the data processing device may include a processor, a memory, a communication interface, and a communication bus in the embodiments of this disclosure.

[0120] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0121] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used to interconnect devices within the data processing equipment, as well as interfaces used to interconnect the data processing equipment with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0122] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0123] The processor can be a general-purpose processor, which can call a data processing program stored in memory and execute the data processing method provided in the embodiments of this disclosure. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the data processing program is called can be referred to in various embodiments of the data processing method of this disclosure, and will not be repeated here.

[0124] Those skilled in the art will understand that the hardware structure shown in FIG6 does not constitute a limitation of this disclosure and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0125] Fourthly, this disclosure provides some embodiments of a vehicle, including a vehicle body and the data processing device provided in the third aspect above.

[0126] Fifthly, embodiments of this disclosure also provide a computer-readable storage medium.

[0127] This disclosure provides a readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the data processing method described above.

[0128] The methods implemented when the data processing program is executed can be referred to in various embodiments of the data processing method disclosed herein, and will not be repeated here.

[0129] Sixthly, some embodiments of this disclosure provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the data processing method provided in the foregoing method embodiments.

[0130] It should be noted that the sequence numbers of the above-mentioned embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0131] The terms "comprising" and "having," and any variations thereof, in this disclosure, claims, and accompanying drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequential order, nor do they limit "first," "second," and "third" to different types.

[0132] In the description of embodiments in this disclosure, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in this disclosure should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0133] In the description of the embodiments of this disclosure, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The word "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this disclosure, "multiple" means two or more.

[0134] In some processes described in the embodiments of this disclosure, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this disclosure, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this disclosure.

[0136] The above are merely preferred embodiments of this disclosure and do not limit the patent scope of this disclosure. Any equivalent structural or procedural transformations made using the content of this disclosure and its drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this disclosure.

Claims

1. A data processing method, comprising: Acquire millimeter-wave radar point voxel data corresponding to millimeter-wave radar data; The millimeter-wave radar point voxel data is encoded to generate target voxel sequence features; Multi-level linear computation is performed on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features; The autocorrelation matrix is ​​obtained by calculating the query sequence features and the key sequence features based on the attention mechanism; as well as The autocorrelation matrix and the value sequence features are encoded, and the encoding results are residually concatenated with the target voxel sequence features to generate autocorrelation enhancement sequence features for millimeter-wave radar.

2. The data processing method as described in claim 1, further comprising, after the step of generating the millimeter-wave radar autocorrelation enhancement sequence features: Based on the voxel coordinates corresponding to the autocorrelation enhancement sequence features and the sensing range of the millimeter-wave radar's bird's-eye view, the index coordinates are determined. as well as Based on the index coordinates, the autocorrelation enhancement sequence features are mapped to the bird's-eye view features of the millimeter-wave radar.

3. The data processing method as described in claim 2, wherein, The steps for determining the perception range of the bird's-eye view include: Obtain the sensing range and voxel length of the millimeter-wave radar point cloud; Based on the voxel length, the perceptual grid size of the bird's-eye view is determined; and The perception range of the bird's-eye view is determined by comparing the perception range of the point cloud with the perception grid size of the bird's-eye view.

4. The data processing method as described in claim 3, wherein, The step of mapping the autocorrelation enhancement sequence features to the bird's-eye view features of the millimeter-wave radar based on the index coordinates includes: Based on the index coordinates, the autocorrelation enhancement sequence features are assigned to the corresponding bird's-eye view perception grid to obtain the bird's-eye view features of the millimeter-wave radar.

5. The data processing method as described in claim 2, wherein, The formula for calculating the index coordinates is: coords = r·vy + vx In the formula, coords represents the index coordinates, r represents the perception range of the bird's-eye view, vy represents the vertical axis coordinate of the voxel, and vx represents the horizontal axis coordinate of the voxel.

6. The data processing method as described in claim 2 or 5, further comprising, after the step of obtaining the millimeter-wave radar bird's-eye view features: Acquire the LiDAR bird's-eye view features corresponding to the LiDAR data and the camera bird's-eye view features corresponding to the camera image data; The feature dimensions of the millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features, and the camera bird's-eye view features are transformed respectively, and all the transformed bird's-eye view features are stitched together to obtain the stitched features; The spliced ​​features were activated using different activation functions to obtain three different sets of learnable attention weights. Each set of learnable attention weights and their corresponding bird's-eye view features are weighted separately to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features, and the target camera bird's-eye view features; and The bird's-eye view features of the target millimeter-wave radar, the target lidar, and the target camera are stitched together to generate a fused feature.

7. The data processing method as described in claim 6, wherein, The feature dimension transformation of the millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features, and the camera bird's-eye view features includes: The features of the millimeter-wave radar bird's-eye view are sequentially convolved, batch normalized, and activated to obtain the transformed millimeter-wave radar bird's-eye view features. The LiDAR bird's-eye view features are sequentially subjected to convolution, batch normalization, and activation operations to obtain the transformed LiDAR bird's-eye view features; and The camera bird's-eye view features are sequentially subjected to convolution, batch normalization, and activation operations to obtain the transformed camera bird's-eye view features.

8. The data processing method as described in claim 1, wherein, The encoding of the autocorrelation matrix and the value sequence features includes: The autocorrelation matrix and the value sequence features are encoded using a multi-head self-attention mechanism.

9. The data processing method as described in claim 1, wherein, The calculation of the autocorrelation matrix based on the attention mechanism for the query sequence features and the key sequence features includes: Based on the activation function, the query sequence features, the transpose of the key sequence features, and the feature channel dimension of the key sequence features, the autocorrelation matrix is ​​obtained.

10. The data processing method as described in claim 9, wherein, The formula for calculating the autocorrelation matrix is ​​as follows: In the formula, M auto Let K represent the autocorrelation matrix, Q represent the query sequence features, and K represent the autocorrelation matrix. T C represents the transpose of the key sequence features. K The feature channel dimension represents the key sequence features.

11. The data processing method as described in claim 1, wherein, The expression for the autocorrelation enhancement sequence features of the millimeter-wave radar is as follows: In the formula, This represents the autocorrelation enhancement sequence features of millimeter-wave radar, where MultiHeadAttention represents the multi-head attention module, and M... auto Let V represent the autocorrelation matrix, V represent the value sequence features, head represent the number of heads in the multi-head attention module, and f represent the autocorrelation matrix. radar This indicates the characteristics of the target voxel sequence.

12. The data processing method as described in claim 1, wherein, The acquisition of millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data includes: Obtain the total number of millimeter-wave radar points collected and the information contained in each millimeter-wave radar point; Based on the total number of millimeter-wave radar points and the information contained in each millimeter-wave radar point, a point set with a total number and information dimension is generated; and The point set is voxelized to obtain millimeter-wave radar point voxel data corresponding to the millimeter-wave radar data.

13. A data processing apparatus, comprising an autocorrelation point enhancement module, for: Acquire millimeter-wave radar point voxel data corresponding to millimeter-wave radar data; The millimeter-wave radar point voxel data is encoded to generate target voxel sequence features; Multi-level linear computation is performed on the target voxel sequence features to generate query sequence features, key sequence features, and value sequence features; The autocorrelation matrix is ​​obtained by calculating the query sequence features and the key sequence features based on the attention mechanism; as well as The autocorrelation matrix and the value sequence features are encoded, and the encoding results are residually concatenated with the target voxel sequence features to generate autocorrelation enhancement sequence features for millimeter-wave radar.

14. The data processing apparatus of claim 13, further comprising an adaptive module for: Acquire the LiDAR bird's-eye view features corresponding to the LiDAR data and the camera bird's-eye view features corresponding to the camera image data; The feature dimensions of the millimeter-wave radar bird's-eye view features, the lidar bird's-eye view features, and the camera bird's-eye view features are transformed respectively, and all the transformed bird's-eye view features are stitched together to obtain the stitched features; The spliced ​​features were activated using different activation functions to obtain three different sets of learnable attention weights. Each set of learnable attention weights and their corresponding bird's-eye view features are weighted separately to obtain the target millimeter-wave radar bird's-eye view features, the target lidar bird's-eye view features, and the target camera bird's-eye view features; and The bird's-eye view features of the target millimeter-wave radar, the target lidar, and the target camera are stitched together to generate a fused feature.

15. The data processing apparatus of claim 13, wherein the adaptability module is further configured to: The features of the millimeter-wave radar bird's-eye view are sequentially convolved, batch normalized, and activated to obtain the transformed millimeter-wave radar bird's-eye view features. The LiDAR bird's-eye view features are sequentially subjected to convolution, batch normalization, and activation operations to obtain the transformed LiDAR bird's-eye view features; and The camera bird's-eye view features are sequentially subjected to convolution, batch normalization, and activation operations to obtain the transformed camera bird's-eye view features.

16. The data processing apparatus of claim 13, wherein the autocorrelation point enhancement module is used for: The autocorrelation matrix and the value sequence features are encoded using a multi-head self-attention mechanism.

17. A data processing apparatus comprising a processor, a memory, and a data processing program stored in the memory and executable by the processor, wherein the data processing program, when executed by the processor, implements the steps of the data processing method as claimed in any one of claims 1 to 12.

18. A vehicle comprising a vehicle body and the data processing device of claim 17.

19. A computer-readable storage medium storing a data processing program thereon, wherein the data processing program, when executed by a processor, implements the steps of the data processing method as claimed in any one of claims 1 to 12.

20. A computer program product comprising computer instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 12.

Images