Semantic segmentation method, semantic segmentation system, and vehicle-mounted terminal

Abstract

A semantic segmentation method, comprising: respectively extracting voxel features and image features on the basis of initial point cloud data and image data of a target scene (S100); obtaining first point-wise features and second point-wise features on the basis of the extraction results (S200), so as to further obtain a geometric fusion feature and a semantic fusion feature (S300); performing modeling to obtain a semantic segmentation model (S400); and then performing semantic segmentation on the target scene by means of the semantic segmentation model (S500). The present application improves the continuity and completeness of features, and can retain spatial structure information and also enhance the depth of semantic understanding, thereby facilitating the capture of global context features.

Description

A semantic segmentation method, a semantic segmentation system, and an in-vehicle terminal Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a semantic segmentation method, a semantic segmentation system, and an in-vehicle terminal. Background Technology

[0002] Semantic segmentation is one of the important tasks in the environmental perception stage of autonomous driving technology. It aims to understand and label the semantic information of different regions of the scene (images, point clouds, etc.) acquired by sensors, and is the foundation for visual tasks such as target recognition and classification.

[0003] One existing semantic segmentation method loses some features due to discontinuous projected views; another method has limited ability to extract local features and cannot capture global contextual features well for point clouds in large scenes. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a semantic segmentation method, a semantic segmentation system, and an in-vehicle terminal that can guarantee the integrity and continuity of feature extraction and effectively capture global contextual features.

[0005] Specifically, this application provides a semantic segmentation method, comprising the following steps: acquiring initial point cloud data and image data of a target scene; extracting voxel features based on the initial point cloud data and extracting image features based on the image data; mapping the voxel features and image features to the initial point cloud data to obtain first point-by-point features and second point-by-point features; performing geometric fusion processing based on the first point-by-point features and second point-by-point features to obtain geometric fusion features, and performing semantic fusion processing based on the voxel features and image features to obtain semantic fusion features; modeling based on the geometric fusion features and semantic fusion features to obtain a semantic segmentation model; and acquiring target point cloud data to be segmented and inputting the target point cloud data into the semantic segmentation model to perform semantic segmentation of the target scene.

[0006] In the above technical solution, the continuity and completeness of features are improved by comprehensively utilizing image data and point cloud data. Through geometric and semantic fusion of voxel features and image features, not only is spatial structural information preserved, but the depth of semantic understanding is also enhanced. This dual fusion can better capture features in complex environments and helps capture global contextual features. Furthermore, since the model is built based on the fusion of the two feature data, semantic consistency between point cloud data and image data can be better guaranteed during semantic segmentation, reducing errors caused by different data sources.

[0007] Furthermore, the extraction of voxel features includes: segmenting the initial point cloud data based on the quantization step size, and obtaining multiple sets of non-empty voxels based on the segmented initial point cloud data; and integrating the multiple sets of non-empty voxels into a point cloud space for rasterization, and extracting the voxel features of each set of non-empty voxels by superimposing sparse convolution and sparse deconvolution.

[0008] In the above technical solution, segmenting the initial point cloud data based on the quantization step size enables refined processing of the point cloud data, effectively reducing data complexity and making subsequent feature extraction more efficient and accurate. Extracting multiple sets of non-empty voxels allows focusing on important spatial regions, helping to reduce the waste of computing resources and improve processing speed. The selection of non-empty voxels allows the model to focus on regions containing actual information, increasing the effectiveness of feature extraction. Integrating multiple sets of non-empty voxels into a single point cloud space for rasterization helps to unify the data format and structure, facilitating subsequent operations. By superimposing sparse convolution and sparse deconvolution to extract voxel features, local and global features can be better captured while maintaining computational efficiency and preserving spatial information, enabling the model to more effectively understand and segment targets in complex scenes. The design of this method makes it suitable for processing large-scale point cloud data. The characteristics of sparse convolution allow the model to maintain efficiency under large-scale data, thereby improving the practicality and applicability of the system.

[0009] Furthermore, the extraction of image features includes: performing nonlinear projection and downsampling operations on the image data through an image backbone to extract image features.

[0010] In the above technical solution, when the image backbone performs nonlinear projection and downsampling operations, it actually extracts features through different levels of convolutional and pooling layers. This multi-level feature extraction can capture a variety of features from low to high level, including edges, textures, shapes, and object contours. Nonlinear projection can introduce complex nonlinear feature mapping relationships, making the features extracted from the image more expressive. Downsampling operations can effectively reduce the size of the feature map, thereby reducing the amount of computation and memory requirements, and improving the efficiency of feature extraction.

[0011] Furthermore, despite the downsampling operation, the design of image backbone networks typically preserves important spatial structural information by using appropriate strides and retaining a certain spatial resolution, which helps to better maintain spatial continuity and consistency during semantic segmentation.

[0012] Furthermore, obtaining the first point-by-point feature and the second point-by-point feature includes: using the voxel feature as the first point-by-point feature; and obtaining point-by-point image coordinates, and performing bilinear interpolation on the image feature based on the initial point cloud data using the point-by-point image coordinates to obtain the second point-by-point feature.

[0013] In the above technical solution, bilinear interpolation is performed on image features based on point-by-point image coordinates. This allows for the selective extraction of image features at different scales and resolutions. This interpolation method effectively reduces errors caused by spatial discretization, thereby improving the representational ability of features. Furthermore, through high-quality point-by-point feature acquisition, the semantic segmentation model can more accurately understand the features of each point and its surrounding environment, thus improving segmentation accuracy. In addition, bilinear interpolation can effectively handle feature information at different scales, adapting to various changes in the scene, such as changes in viewpoint and object size, enhancing the model's adaptability.

[0014] Furthermore, obtaining the geometric fusion feature includes: splicing the first point-by-point feature and the second point-by-point feature to obtain the geometric fusion feature.

[0015] In the above technical solution, by splicing the first point-by-point feature (voxel feature) and the second point-by-point feature (image feature), information from different data sources can be fully integrated, enabling the model to more comprehensively understand the geometric characteristics and semantic information of the target scene.

[0016] Furthermore, the semantic fusion feature includes at least a first semantic feature and a second semantic feature; obtaining the semantic fusion feature includes: predicting a first intermediate segmentation feature based on the voxel feature, and normalizing each first intermediate segmentation feature to obtain a first semantic feature; and predicting a second intermediate segmentation feature based on image features, and performing a rearrangement operation and normalization on the second intermediate segmentation feature to obtain a second semantic feature.

[0017] In the above technical solution, the normalization processing of both the first and second intermediate segmentation features can effectively reduce the differences between features, making the features more stable and consistent in subsequent model processing, which helps to improve the training efficiency and generalization ability of the model; the first and second semantic features generated based on voxels and image features respectively can provide rich contextual information; the rearrangement operation of the second intermediate segmentation features enables the image features to be integrated into the segmentation task in the most efficient way.

[0018] Furthermore, obtaining the semantic segmentation model includes: projecting the geometric fusion features, the first semantic feature, and the second semantic feature into a multi-dimensional space to obtain point-by-point projection features, and concatenating the point-by-point projection features to obtain typical features; performing a first modeling based on the semantic relationship of the typical features through multi-head self-attention, and decomposing the multi-head self-attention into single-head self-attention to obtain semantic embedding features; and performing a second modeling based on the semantic relationship of the point-by-point projection features and the semantic embedding features through multi-head cross-attention to obtain the semantic segmentation model.

[0019] In the above technical solutions, the mapping of multi-dimensional space enables different features to more effectively capture and express complex semantic relationships in a higher-dimensional space; the generation of typical features enables the model to more comprehensively understand all aspects of the scene; the multi-head self-attention mechanism can observe the relationship between different features in parallel, improving the model's contextual understanding ability; decomposing multi-head self-attention into single-head self-attention not only reduces computational complexity but also improves the model's inference speed and efficiency; single-head self-attention, while maintaining computational accuracy, is more suitable for real-time application scenarios; multi-head cross-attention can process multi-source features in parallel, improving the accuracy of semantic segmentation.

[0020] Furthermore, the entire modeling process involves multiple hierarchical processes, including multi-dimensional spatial projection, stitching, self-attention, and cross-attention modeling, forming a multi-layered semantic understanding system. This system can deeply understand the semantic structure of complex scenes and significantly improve segmentation quality.

[0021] Based on the same concept, this application also provides a semantic segmentation system, which includes at least an intelligent driving domain controller; the intelligent driving domain controller is used to obtain a semantic segmentation model based on the initial point cloud data and image data of the acquired target scene, and input the target point cloud data to be segmented into the semantic segmentation model to perform semantic segmentation on the target scene.

[0022] Furthermore, the intelligent driving domain controller includes at least a memory and a processor; the memory is used to store computer instructions for multiple functional layers, each functional layer including at least a feature processing functional layer, a modeling functional layer, and a segmentation functional layer, each functional layer including one or more functional modules; the processor communicates with the memory via a bus to execute the computer instructions for each of the multiple functional layers stored in the memory.

[0023] Furthermore, the feature processing functional layer includes at least a feature extraction functional module, a feature mapping functional module, and a feature fusion functional module; the feature extraction functional module includes computer instructions for extracting voxel features based on initial point cloud data of the target scene and extracting image features based on image data of the target scene; wherein the feature extraction functional module provides input data to the feature mapping functional module; the feature mapping functional module includes computer instructions for mapping the voxel features and image features to the initial point cloud data to obtain first point-by-point features and second point-by-point features; wherein the feature mapping functional module provides input data to the feature fusion functional module; the feature fusion functional module includes computer instructions for performing geometric fusion processing based on the first point-by-point features and second point-by-point features to obtain geometric fusion features, and performing semantic fusion processing based on the voxel features and image features to obtain semantic fusion features.

[0024] Furthermore, the modeling function layer includes at least a modeling function module, and the feature fusion function module provides input data to the modeling function module; the modeling function module includes computer instructions for performing modeling based on the geometric fusion features and semantic fusion features to obtain a semantic segmentation model.

[0025] Furthermore, the segmentation function layer includes at least a semantic segmentation function module, and the modeling function module outputs a semantic segmentation model to the semantic segmentation function module; the semantic segmentation function module includes computer instructions for acquiring target point cloud data to be segmented and inputting the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

[0026] Furthermore, the semantic segmentation system also includes a cloud server, which is used to acquire initial point cloud data and image data of the target scene, and send the initial point cloud data and image data as input data to the feature extraction function module.

[0027] In the aforementioned technical solution, the combination of image data and point cloud data significantly improves the continuity and completeness of features. Specifically, by performing geometric and semantic fusion on voxel features and image features, spatial structural information is preserved while enhancing the depth of semantic understanding. This dual fusion mechanism can more effectively capture features in complex environments, especially global context features, thereby improving the model's ability to understand complex scenes. Furthermore, since the model is built based on the fusion of two types of feature data, it can better ensure the semantic consistency between point cloud data and image data during semantic segmentation, reducing errors caused by differences in data sources. This approach not only improves the accuracy of segmentation results but also enhances the model's robustness and adaptability in different environments.

[0028] Based on the same concept, this application also provides an in-vehicle terminal, which is configured with at least a semantic segmentation system to perform semantic segmentation on a target scene by executing the semantic segmentation method through the semantic segmentation system.

[0029] Compared with the prior art, the beneficial effects of this application are as follows:

[0030] This application comprehensively utilizes image data and point cloud data to improve the continuity and completeness of features; it can also effectively preserve spatial structure information, enhance the depth of semantic understanding, and thus achieve effective acquisition of global contextual features. Attached Figure Description

[0031] Figure 1 is a flowchart of the semantic segmentation method described in an embodiment of this application.

[0032] Figure 2 is a framework diagram of the semantic segmentation system described in an embodiment of this application.

[0033] Figure 3 is a framework diagram of the intelligent driving domain controller described in an embodiment of this application.

[0034] Figure 4 is a framework diagram of the memory described in an embodiment of this application.

[0035] Figure 5 is a schematic diagram of the connection between the cloud server and the storage device according to an embodiment of this application.

[0036] Figure 6 is a frame diagram of the vehicle terminal described in an embodiment of this application. Detailed Implementation

[0037] The semantic segmentation method, semantic segmentation system, and vehicle terminal of this application will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0038] Please refer to Figure 1. This application embodiment provides a semantic segmentation method, including the following steps S100-S500.

[0039] Step S100: Obtain initial point cloud data and image data of the target scene, extract voxel features based on the initial point cloud data, and extract image features based on the image data.

[0040] The nuScenes dataset consists of 28,130 training samples, 6,019 validation samples, and 6,008 test samples [3, 11]. Each nuScenes sample contains a relatively sparse 32 bundles of point cloud (i.e., the initial point cloud data) and RGB images captured by 6 cameras (i.e., the image data): front, front left, front right, back, back left, and back right.

[0041] It should be noted that the above values ​​are merely illustrative examples, and those skilled in the art can set and select the values ​​according to actual needs; the nuScenes dataset is a public, large-scale dataset for autonomous driving developed by the Motional team.

[0042] In some embodiments, the extraction of voxel features includes: segmenting the initial point cloud data based on a quantization step size, and obtaining multiple sets of non-empty voxels based on the segmented initial point cloud data.

[0043] In some implementations, a general sparse 3D U_Net[9,35] is used for voxel-based LiDAR feature extraction. The initial point cloud data is radar point cloud data. The input radar point cloud data is first segmented by a quantization step size d, and then grouped into non-empty voxels by the point-by-point initial features of local points within the average voxel.

[0044] The 3D U-Net[9,35] is a deep learning network architecture for three-dimensional image segmentation, which is an extension of the traditional U-Net; its convolution, pooling and upsampling operations are all performed in three-dimensional space, which enables it to effectively utilize the spatial information of three-dimensional images.

[0045] Here, these non-empty voxels Vin∈Rwexl constitute the sparse tensor input of the point cloud backbone, for learning the voxel feature V∈RNvaxelXCvoxel. Vin∈RwexlX indicates that the input data Vin is a real tensor of shape (width, length); V∈RNvaxelXCvoxel indicates that the voxel feature V is a real tensor of shape (Nvaxel, Cvoxel); Nvaxel is the number of voxels, and Cvoxel is the feature dimension of each voxel.

[0046] Furthermore, multiple groups of non-empty voxels are integrated into a single point cloud space for rasterization, and voxel features of each group of non-empty voxels are extracted by superimposing sparse convolution and sparse deconvolution.

[0047] Furthermore, in some implementations, due to the advantages of 3D voxels, the entire point cloud space is first rasterized to achieve data structuring; these spatial voxel grids have a resolution of L×W×H. Voxel features of each non-empty voxel are extracted by superimposing sparse convolutions and sparse deconvolutions, where the initial features of each voxel are simply calculated as the average of the coordinates of points within each voxel in the LiDAR coordinate system. By calculating the mean of points within each non-empty voxel grid, a new point cloud with point features (i.e., voxel features) is formed. Because the size of a voxel is much smaller than the entire 3D space (e.g., 5cm×5cm×10cm in the method), the voxelized point cloud can not only be processed more efficiently using a sparse convolution-based backbone, but is also approximately equivalent to the original point cloud in 3D object detection.

[0048] Specifically, a voxel encoding / decoding architecture similar to U-net is used to perform 3D sparse convolution (i.e., the sparse convolution) and 3D sparse deconvolution (i.e., the sparse deconvolution) on the obtained sparse voxels. The downsampling process consists of four sparse convolutional modules with dimensions of 16-32-64-64. Each sparse convolutional layer uses a kernel size of 3*3*3, a stride of 2, and a downsampling factor of 8. Each sparse convolutional layer is followed by two sub-stream convolutional layers with a kernel size of 3*3*3 and a stride of 1.

[0049] In some embodiments, the extraction of image features includes: performing nonlinear projection and downsampling operations on the image data through an image backbone to extract image features.

[0050] In some implementations, to enhance LiDAR characteristics with jointly optimized camera features, a trainable image backbone (HRNet-w481441 by default) is used to nonlinearly project image data Xin as X∈RNom×Cimg×HxW, where RNom represents the number of feature maps, HxW represents the downsampled shape (including height H and width W), and an additional channel Cimg is added. The point cloud and image backbone can be flexibly selected from various mature networks. The two backbones in the entire 3D segmentation model are jointly optimized, learning relevant semantic feature representations from different modalities. The expressive voxel features V and image features X ensure the fusion of cross-modal features.

[0051] Among them, HRNet-w48 is a high-resolution network composed of multiple branches, which can extract features at different resolutions and is very suitable for tasks such as image segmentation; the trainable representation means that the parameters of this backbone network can be adjusted and optimized through backpropagation.

[0052] The “non-linear projection” here refers to the processing of the input image through a backbone network to generate new feature representations. This processing usually involves operations such as multi-layer convolution, activation functions, and pooling.

[0053] Furthermore, after extracting voxel features and image features, the following steps S200 can be performed.

[0054] Step S200: Map the voxel features and image features to the initial point cloud data to obtain the first point-by-point features and the second point-by-point features.

[0055] In some implementations, point-centered segmentation excitation maps voxel features V and image features X to points in Pin (i.e., the initial point cloud data) using the geometric association of LiDAR and multiple cameras, as point-by-point LiDAR features (i.e., the first point-by-point features) Flidar∈RNpoint×Cvoxel and point-by-point camera features (i.e., the second point-by-point features) Fcam∈RNpoint×Cimg; where Npoint represents the number of features.

[0056] In some embodiments, obtaining the first point-by-point feature and the second point-by-point feature includes: using the voxel feature as the first point-by-point feature; and obtaining point-by-point image coordinates, and performing bilinear interpolation on the image feature based on the initial point cloud data using the point-by-point image coordinates to obtain the second point-by-point feature.

[0057] In some implementations, voxel features V are used as point-by-point lidar features Flidar = [flidar,i]i = Npointi = 1 ∈ RNpoint × Cvoxel to enhance point-by-point fusion and segmentation.

[0058] Furthermore, given a point (i.e., any point in the initial point cloud data), its point features are interpolated from its three nearest neighbor voxels; for the i-th point (xi, yi, zi) in the LiDAR with LiDAR functionality flidar,i, bilinear interpolation I, such as 1, is performed on the image features X[ci]∈RCimg×H×W within the ci-th local camera using the known point-by-point image coordinates (ci, ui, vi). Zeros are temporarily filled into the camera FOV outside those points, which allows each point to be decorated with the Cimg-dimensional feature fcam,i. To distinguish external points, a binary mask B is set for the Npoint elements, setting 0 for them and 1 otherwise.

[0059] in:

[0060] ;

[0061] Furthermore, after obtaining the first point-by-point feature and the second point-by-point feature, the following steps S300 can be performed.

[0062] Step S300: Perform geometric fusion processing based on the first point-by-point feature and the second point-by-point feature to obtain geometric fusion features, and perform semantic fusion processing based on the voxel feature and image feature to obtain semantic fusion features.

[0063] In some embodiments, obtaining geometric fusion features includes: splicing the first point-by-point feature and the second point-by-point feature to obtain geometric fusion features.

[0064] In some implementations, by using a fully connected layer to represent the point-by-point LiDAR features Flidar and the point-by-point camera features Fcam, flidar,i and fcam,i are projected as Cint dimensions and concatenated together (“©”) for subsequent learnable fusion using another MLP, whose Cgfused output channel fgfused,i = MLP(Flidar(flidar,i)©Fcam(fcam,i)) and Fgfused = [fgfused,i]i = Npointi = 1 ∈ RNpoint × Cgfused. GF-Phase combines the multimodal features Fgfused primarily with geometric cues, referred to as the geometry-based feature fusion stage.

[0065] In some embodiments, the semantic fusion features include at least a first semantic feature and a second semantic feature; obtaining the semantic fusion features includes: predicting a first intermediate segmentation feature based on the voxel features, and normalizing each first intermediate segmentation feature to obtain the first semantic feature.

[0066] In some implementations, the LiDAR Semantic Feature Aggregation Module (LiDAR SFAM) starts with voxel features V and ends with the aggregation of LiDAR semantic embeddings in NclsElidar. For Ncls categories, assume Elidar∈RNcls×Cvoxel and a distribution matrix Dlidar∈(0, 1)Ncls×Nvoxel; where (0, 1) indicates that the values ​​of the elements are between 0 and 1, Ncls represents the number of categories, and Nvoxel represents the number of voxels.

[0067] A closer D[j, i] to 1 indicates that the i-th voxel is more likely to belong to the j-th class than other voxels, and therefore contributes more to describing this class in Elidar[j]. Otherwise, the i-th voxel should contribute less to Elidar[j]. Elidar can be computed via matrix multiplication Elidar = DlidarV. Two more steps are needed to compute Dlidar: first, an auxiliary voxel segmentation head Hvoxel (Hvoxel represents the height of each voxel) based on MLP is used to predict the intermediate segmentation D′lidar∈RNvoxel×Ncls on the voxel feature V (i.e., predicting the first intermediate segmentation feature). Then, spatial softmax is performed among the voxels to normalize the value of each class to (0, 1) (i.e., normalizing each first intermediate segmentation feature).

[0068] in:

[0069] ;

[0070] Furthermore, based on image features, a second intermediate segmentation feature is predicted, and the second intermediate segmentation feature is rearranged and normalized to obtain a second semantic feature.

[0071] In some implementations, for more accurate Dlidar, D′lidar is explicitly guided by the use of voxel segmentation supervision.

[0072] The Camera Semantic Feature Aggregation Module (Camera sfm) uses another distribution matrix Dcam∈(0, 1)Ncls×Npixel and another auxiliary image segmentation head Himg implemented by a simple FCN head to aggregate Ecam∈RNcls×Cimg from the image feature map X∈RNcam×Cimg×H×W. The difference is that Camera sfm designs the image feature map of all Npixel pixels of the local camera, where Npixel is Ncam×H×W. Himg is first predicted by Himg(X) to form the intermediate segment D′img∈RNcam×Ncls×H×W. Here, Npixel represents the number of pixels; RNcam represents the number of images or cameras in the data batch; Cimg represents the number of channels of the image; and H×W represents the height and width of the image.

[0073] It should be noted that, through transpose and reshaping operations, D′img and X are rearranged into D′cam∈RNcls×Npixel and X′∈RNpixel×Cimg. The final camera semantic embedding Ecam is the matrix product of Dcam and X′, where the normalized Dcam is obtained by applying spatial softmax to D′cam.

[0074] in:

[0075] ;

[0076] Furthermore, after obtaining the geometric fusion features and semantic fusion features, the following steps S400 can be performed.

[0077] Step S400: Model based on the geometric fusion features and semantic fusion features to obtain a semantic segmentation model.

[0078] In some embodiments, obtaining the semantic segmentation model includes: projecting the geometric fusion feature, the first semantic feature, and the second semantic feature onto a multidimensional space to obtain point-by-point projection features, and stitching the point-by-point projection features to obtain typical features.

[0079] In some implementations, given a point-by-point geometrically fused feature Fgfused, a class-by-class semantic embedding Elidar (i.e., the first semantic feature), and an input Ecam (i.e., the second semantic feature), SFFM first projects them into the Csfused dimensional space as Fproj = Proj1(Fgfused) and E = Proj2(Elidar) © Proj3(Ecam), respectively. Before stacking K grayscale blocks in SFFM, the projected multimodal semantic embeddings are concatenated to form E∈R2Ncls×Csfused, which can be viewed as a dictionary describing the typical features of each class from the perspective of LiDAR and camera.

[0080] A semantic segmentation model is obtained by performing a first modeling based on the semantic relationships of the typical features through multi-head self-attention, and then decomposing the multi-head self-attention into single-head self-attention; and a second modeling is obtained based on the semantic relationships of the point-by-point projection features and the semantic embedding features through multi-head cross-attention.

[0081] In some implementations, within each gray block, multi-head self-attention (MHSA) is used to model the semantic relationships between all 2Ncls classification semantic embeddings, where Norm denotes the LayerNorm operation. Let Cshsa be Csfused / NH, MHSA can be decomposed into NH single-head self-attention (SHSA) operations, where Eh∈R2Ncls×Cshsa is:

[0082] ;

[0083] ;

[0084] ;

[0085] After updating E to Emhsa, the pointwise projection feature Fproj and the semantic embedding feature Emhsa are further fused. Multi-head cross-attention (MHCA) is used for semantic relationship modeling, and a feedforward network (FFN) is used for feature embedding. Wherein:

[0086] ;

[0087] ;

[0088] It should be noted that Ncls is preferably 17, but those skilled in the art can also set it to other values ​​according to actual application requirements.

[0089] Furthermore, after obtaining the semantic segmentation model, the following steps S500 can be performed.

[0090] Step S500: Obtain the target point cloud data to be segmented, and input the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

[0091] In some embodiments, a semantic segmentation model assigns each pixel in the target scene image to a specific category label, such as road, sky, or grass, enabling the vehicle to better understand its environment by recognizing different objects on the road. The output of the semantic segmentation model can be applied to modules such as environmental perception, path planning, blind spot detection, lane keeping assist, and automatic parking. By outputting the semantic segmentation results to the corresponding memory functional layers of these intelligent assistance modules, further driver assistance functions can be achieved.

[0092] Please refer to Figure 2. This application also provides a semantic segmentation system 1, which includes at least an intelligent driving domain controller 10. The intelligent driving domain controller 10 is used to obtain a semantic segmentation model based on the initial point cloud data and image data of the target scene, and input the target point cloud data to be segmented into the semantic segmentation model to perform semantic segmentation on the target scene.

[0093] Furthermore, referring to Figure 3, the intelligent driving domain controller 10 includes at least a memory 101 and a processor 102; the memory 101 is used to store computer instructions for multiple functional layers, and the functional layers include at least a feature processing functional layer 104, a modeling functional layer 105, and a segmentation functional layer 106 (as shown in Figure 4), each functional layer including one or more functional modules; the processor 102 communicates with the memory 101 via a bus and is used to execute the computer instructions for each of the multiple functional layers stored in the memory 101.

[0094] The memory 101 includes, but is not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), and CD-ROM (Compact Disc Read-Only Memory); the processor 102 includes a central processing unit or a device or module with processing capabilities.

[0095] Furthermore, the feature processing functional layer 104 includes at least a feature extraction functional module 1041, a feature mapping functional module 1042, and a feature fusion functional module 1043. The feature extraction functional module 1041 includes computer instructions for extracting voxel features based on initial point cloud data of the target scene and extracting image features based on image data of the target scene. The feature extraction functional module 1041 provides input data to the feature mapping functional module 1042. The feature mapping functional module 1042 includes computer instructions for mapping the voxel features and image features to the initial point cloud data to obtain first point-by-point features and second point-by-point features. The feature mapping functional module 1042 provides input data to the feature fusion functional module 1043. The feature fusion functional module 1043 includes computer instructions for performing geometric fusion processing based on the first point-by-point features and second point-by-point features to obtain geometric fusion features, and performing semantic fusion processing based on the voxel features and image features to obtain semantic fusion features.

[0096] Furthermore, the modeling function layer 105 includes at least a modeling function module 1051, and the feature fusion function module 1043 provides input data to the modeling function module 1051; the modeling function module 1051 includes computer instructions for modeling based on the geometric fusion features and semantic fusion features to obtain a semantic segmentation model.

[0097] Furthermore, the segmentation function layer 106 includes at least a semantic segmentation function module 1061, and the modeling function module outputs a semantic segmentation model to the semantic segmentation function module 1061; the semantic segmentation function module 1061 includes computer instructions for acquiring target point cloud data to be segmented and inputting the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

[0098] Furthermore, the semantic segmentation system 1 also includes a cloud server 20, which is used to acquire initial point cloud data and image data of the target scene, and send the initial point cloud data and image data as input data to the feature extraction function module 1041 (as shown in Figure 5).

[0099] The cloud server acquires the nuScenes dataset. The data in this dataset is collected using a vehicle equipped with multiple sensors, including at least a camera array and a LiDAR. The camera array is equipped with six high-resolution cameras, covering a 360-degree field of view around the vehicle to capture rich visual information. The LiDAR is a 32-line LiDAR that can provide three-dimensional point cloud data of the vehicle's surrounding environment for perceiving the position and shape of objects.

[0100] Furthermore, referring to Figure 6, this application also provides an in-vehicle terminal 2, which is configured with at least a semantic segmentation system 1 to perform semantic segmentation on the target scene by executing the semantic segmentation method through the semantic segmentation system 1.

[0101] In summary, this application provides a semantic segmentation method, a semantic segmentation system, and an in-vehicle terminal. First, voxel features and image features are extracted from initial point cloud data and image data of the target scene, respectively. Then, first point-by-point features and second point-by-point features are obtained based on the voxel features and image features to further obtain geometric fusion features and semantic fusion features. Next, a semantic segmentation model is obtained based on the geometric fusion features and semantic fusion features. Finally, the target point cloud data to be segmented is input into the semantic segmentation model to perform semantic segmentation of the target scene. This application improves the continuity and completeness of features; it not only preserves spatial structure information but also enhances the depth of semantic understanding, helping to capture global contextual features.

[0102] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0103] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0105] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0106] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0107] Although the description of this application has been made in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.

Claims

1. A semantic segmentation method, comprising the following steps: Acquire initial point cloud data and image data of the target scene, extract voxel features based on the initial point cloud data, and extract image features based on the image data (S100). The voxel features and image features are mapped to the initial point cloud data to obtain the first point-by-point features and the second point-by-point features (S200). Geometric fusion processing is performed based on the first point-by-point feature and the second point-by-point feature to obtain geometric fusion features, and semantic fusion processing is performed based on the voxel feature and image feature to obtain semantic fusion features (S300). Based on the geometric fusion features and semantic fusion features, a semantic segmentation model is obtained (S400). In addition, the target point cloud data to be segmented is obtained, and the target point cloud data is input into the semantic segmentation model to perform semantic segmentation on the target scene (S500).

2. The semantic segmentation method according to claim 1, wherein extracting voxel features includes: The initial point cloud data is segmented based on the quantization step size, and multiple sets of non-empty voxels are obtained based on the segmented initial point cloud data. Furthermore, multiple groups of non-empty voxels are integrated into a single point cloud space for rasterization, and voxel features of each group of non-empty voxels are extracted by superimposing sparse convolution and sparse deconvolution.

3. The semantic segmentation method according to claim 2, wherein extracting image features includes: Image features are extracted by performing nonlinear projection and downsampling operations on the image data using an image backbone.

4. The semantic segmentation method according to claim 3, wherein obtaining the first pointwise feature and the second pointwise feature includes: The voxel features are used as the first point-by-point features; Furthermore, point-by-point image coordinates are obtained, and based on the initial point cloud data, bilinear interpolation is performed on the image features using the point-by-point image coordinates to obtain the second point-by-point features.

5. The semantic segmentation method according to claim 4, wherein obtaining the geometric fusion features includes: The first point-by-point feature and the second point-by-point feature are spliced ​​together to obtain the geometric fusion feature.

6. The semantic segmentation method according to claim 5, wherein the semantic fusion features include at least a first semantic feature and a second semantic feature; The obtained semantic fusion features include: Based on the voxel features, predict the first intermediate segmentation features, and normalize each first intermediate segmentation feature to obtain the first semantic features. Furthermore, based on image features, a second intermediate segmentation feature is predicted, and the second intermediate segmentation feature is rearranged and normalized to obtain a second semantic feature.

7. The semantic segmentation method according to claim 6, wherein obtaining the semantic segmentation model comprises: The geometric fusion features, the first semantic features, and the second semantic features are projected onto a multi-dimensional space to obtain point-by-point projection features, and the point-by-point projection features are then stitched together to obtain typical features.

8. The semantic segmentation method according to claim 7, wherein obtaining the semantic segmentation model further includes: The semantic relationships of the typical features are modeled once by multi-head self-attention, and the multi-head self-attention is decomposed into single-head self-attention to obtain semantic embedding features; A semantic segmentation model is obtained by performing secondary modeling based on the semantic relationship between the point-by-point projection features and semantic embedding features through multi-head cross-attention.

9. A semantic segmentation system (1), comprising at least an intelligent driving domain controller (10); the intelligent driving domain controller (10) is used to obtain a semantic segmentation model based on the initial point cloud data and image data of the acquired target scene, and input the target point cloud data to be segmented into the semantic segmentation model to perform semantic segmentation on the target scene.

10. The semantic segmentation system (1) according to claim 9, wherein the intelligent driving domain controller (10) includes at least a memory (101) and a processor (102). The memory (101) is used to store computer instructions for multiple functional layers, the functional layers including at least a feature processing functional layer (104), a modeling functional layer (105) and a segmentation functional layer (106), each functional layer including one or more functional modules; The processor (102) communicates with the memory (101) via a bus (103) to execute computer instructions for each of the multiple functional layers stored in the memory (101).

11. The semantic segmentation system (1) according to claim 10, wherein the feature processing functional layer (104) includes at least a feature extraction functional module (1041), a feature mapping functional module (1042), and a feature fusion functional module (1043). The feature extraction module (1041) includes computer instructions for extracting voxel features from initial point cloud data of the target scene and extracting image features from image data of the target scene; wherein, The feature extraction module (1041) provides input data to the feature mapping module (1042); The feature mapping function module (1042) includes computer instructions for mapping the voxel features and image features to the initial point cloud data to obtain first point-by-point features and second point-by-point features; wherein, the feature mapping function module (1042) provides input data to the feature fusion function module (1043); The feature fusion function module (1043) includes computer instructions for performing geometric fusion processing based on the first point-by-point feature and the second point-by-point feature to obtain geometric fusion features, and performing semantic fusion processing based on the voxel feature and image feature to obtain semantic fusion features.

12. The semantic segmentation system (1) according to claim 11, wherein the modeling functional layer (105) includes at least a modeling functional module (1051), and the feature fusion functional module (1043) provides input data to the modeling functional module (1051); The modeling function module (1051) includes computer instructions for modeling based on the geometric fusion features and semantic fusion features to obtain a semantic segmentation model.

13. The semantic segmentation system (1) according to claim 12, wherein the segmentation function layer (106) includes at least a semantic segmentation function module (1061), and the modeling function module outputs a semantic segmentation model to the semantic segmentation function module (1061). The semantic segmentation function module (1061) includes computer instructions for acquiring target point cloud data to be segmented and inputting the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

14. The semantic segmentation system (1) according to claim 11 further includes a cloud server (20), wherein the cloud server (20) is used to acquire initial point cloud data and image data of the target scene, and send the initial point cloud data and image data as input data to the feature extraction function module (1041).

15. A vehicle-mounted terminal (2) is configured with at least a semantic segmentation system (1) to perform semantic segmentation on a target scene by executing the semantic segmentation method as described in claim 1 through the semantic segmentation system (1).

Images