Multi-modal vectorized mapping method and computing device

Abstract

A multi-modal vectorized mapping method, comprising: segmenting a global map to obtain a local map, the global map being constituted by multi-modal data; extracting a multi-modal feature of the local map; obtaining a local instance vector on the basis of the multi-modal feature, the local instance vector being a data sequence of geometric semantic information of the local map; and obtaining a global vectorized map on the basis of the local instance vector. In this way, in the embodiments of the present invention, the local instance vector is outputted on the basis of the multi-modal feature, automatic extraction and analysis of the geometric semantic information of a map can be performed without annotation, and rolling update of an intelligent driving map can be achieved, thereby helping an intelligent driving vehicle to better recognize the surrounding environment.

Description

A multimodal vectorization mapping method and computing device

[0001] This application claims priority to Chinese Patent Application No. 202411556003.3, filed on November 1, 2024, entitled "A Multimodal Vectorization Graphing Method and Computing Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of intelligent driving technology, and in particular to a multimodal vectorization mapping method and computing device. Background Technology

[0003] In today's era of rapid technological advancement, intelligent driving technology is gradually becoming a reality. Extraction of geometric semantic information is a crucial step in intelligent driving map creation. Geometric semantic information extraction refers to extracting specific geographic information from a map, including lane lines, intersection surfaces, roadside boundaries, and other elements, and using this geographic information to locate, plan routes, and control intelligent driving vehicles. Currently, geometric semantic information extraction requires entity category labeling, making it impossible to achieve continuous updates of geometric semantic information. Summary of the Invention

[0004] This application provides a multimodal vectorization mapping method and computing device that automatically extracts geometric semantic information from maps without requiring entity category labeling of pixels on the map. This enables rolling updates of intelligent driving maps, helping intelligent driving vehicles better identify their surroundings.

[0005] In a first aspect, embodiments of this application provide a multimodal vectorized mapping method, the method comprising: segmenting a global map to obtain a local map, the global map being composed of multimodal data; extracting multimodal features from the local map; obtaining local instance vectors based on the multimodal features; the local instance vectors being a data sequence of geometric semantic information of the local map; and obtaining a global vectorized map based on the local instance vectors, the global vectorized map being an image representing the geometric semantic information of the global map in a vectorized form.

[0006] Therefore, in the method provided in this application embodiment, local instance vectors representing geometric semantic information are obtained based on the multimodal features of the local map. This allows for the extraction of geometric semantic information from the local map without needing to label pixels on the map with entity categories. Furthermore, local instance vectors have a small data volume and high computational efficiency. Using local instance vectors, a global vectorized map can be quickly obtained, enabling rolling updates of the geometric semantic information of the global map, thereby helping intelligent driving vehicles better recognize their surrounding environment.

[0007] In some implementations, obtaining a local instance vector based on multimodal features includes: fusing the multimodal features to obtain a first fused feature; performing deep supervised learning on the first fused feature to obtain a second fused feature; and decoding the first fused feature and the second fused feature to obtain a local instance vector.

[0008] Therefore, the embodiments of this application achieve automated extraction and parsing of geometric semantic information of the obtained map by fusing, supervising learning and decoding multimodal features.

[0009] In some implementations, the method further includes: acquiring multimodal features of surrounding images of a local map; the surrounding images include images above, below, to the left, and to the right of the local map; fusing the multimodal features to obtain a first fused feature, including: using a first cross-attention mechanism to interactively learn the multimodal features of the local map and the multimodal features of the surrounding images to obtain optimized multimodal features; and fusing the optimized multimodal features.

[0010] Therefore, the embodiments of this application, through pseudo-temporal feature fusion, can broaden the prediction perspective, enhance the model's ability to extract semantic and geometric features, improve network recall, and have a strong ability to reproduce roads.

[0011] In some implementations, the method further includes: performing hyperparameter learning on the multimodal features to obtain weighted weights; adaptively updating the weighted weights according to scene changes on the local map; and fusing the multimodal features to obtain a first fused feature, including: merging and encoding the multimodal features according to the weighted weights to obtain the first fused feature.

[0012] Therefore, in this application embodiment, the weighted weights between features are dynamically learned in different scenarios using global features and local features. Different modal features are complementary, and the weighted weights are dynamically updated adaptively according to scene changes on the local map. For noisy data, smaller weights can be adaptively learned to reduce the impact of noise.

[0013] In some implementations, the method further includes: performing dense task learning based on a first fusion feature; determining dense features of a local map, the dense features including semantic segmentation features and / or depth features of the local map; and performing deep supervised learning based on the first fusion feature to obtain a second fusion feature, including: using a second cross-attention mechanism to interactively learn the first fusion feature and the dense feature to obtain the second fusion feature.

[0014] Therefore, the interactive learning of image features and dense features in this application embodiment can promote information exchange between and within instances, thereby further improving the learning effect of the model. The interaction between image features and dense task features enhances the model's representation ability, allowing the model's cross-attention mechanism to focus more on the effective region, accelerating convergence and model representation.

[0015] In some implementations, the local instance vector is obtained by decoding based on the first fusion feature and the second fusion feature, including: the first fusion feature is updated through self-attention mechanism; and the second fusion feature and the self-updated first fusion feature are interactively learned using a third cross-attention mechanism to obtain the local instance vector.

[0016] This facilitates information exchange between and within instances, thereby further improving the learning effect of local instance vectors.

[0017] In some implementations, after the second fused feature and the self-learning updated first fused feature are interactively learned using a third cross-attention mechanism, the method further includes: using a fourth cross-attention mechanism to interactively learn the prior features and the output of the third cross-attention mechanism to obtain local instance vectors, where the prior features are vectorized features of the local map.

[0018] Therefore, in this embodiment of the application, the cross-attention mechanism that combines vectorized features and positional features to form positional features is used, and the positional features provide vectorized prior features; the second feature interaction fusion will further improve the accuracy and robustness of the model, thereby better adapting to various practical application scenarios.

[0019] In some implementations, the multimodal features include scalarized features. Extracting multimodal features from a local map includes: dividing the local map into blocks to obtain multiple block images; using an image feature extraction network with multiple transformers to extract multi-scale semantic information of each block image; and superimposing the multi-scale semantic information of each block image to obtain scalarized features of the local map.

[0020] Therefore, the method provided in this application embodiment can automatically extract scalarized features from multimodal data. The network structure can output feature maps at multiple stages. A feature pyramid is obtained based on the feature maps at multiple stages. The feature pyramid includes multi-scale semantic information. The feature pyramid is upsampled, and multi-scale semantic information is superimposed after the height and width of the image are reduced to a fixed 1 / 2M scale to obtain scalarized features of the scalarized data for use by downstream modules. This reduces the complexity of the model while enhancing the feature extraction capability of the image.

[0021] In some implementations, the multimodal features also include vectorized features. Extracting multimodal features from a local map includes: sampling the coordinates of N points on the local map to determine the coordinates of the N points; determining the categories of the coordinates of the N points; stitching together the coordinates and categories of the N points to obtain vectorized instances; and extracting features from the vectorized instances to obtain the vectorized features of the local map.

[0022] Therefore, the method provided in this application embodiment can automatically extract vectorized geometric attributes such as lane lines, road boundaries, intersection surfaces, pedestrian crossings, and buildings, as well as extract city-level semantic information, thus significantly improving the quality of RoadCode mapping.

[0023] In some implementations, the vectorized features include prior features. Feature extraction is performed on the vectorized instances to obtain the vectorized features of the local map, including: extracting coordinate features and category features from the vectorized instances using a position encoder and a category encoder; and concatenating the coordinate features and category features to obtain prior features.

[0024] Therefore, in the multimodal vectorized mapping method provided in this application embodiment, prior features are used as vectorized features of the local map, and then cross-attention mechanism is used with features of other modalities to strengthen the prior query, so that the query can better obtain features of multimodal and prior information.

[0025] In some implementations, there are multiple local instance vectors. Obtaining a global vectorized map based on the local instance vectors includes: concatenating multiple local instance vectors to obtain a global instance vector; and sparsely sampling the point coordinates of the global instance vectors to obtain a global vectorized map.

[0026] Therefore, the multimodal vectorized mapping method provided in this application performs vectorized stitching on the multimodal global map and resparsely samples the point coordinates of the stitched vectorized instances. This eliminates the need for complex post-processing such as finding the main pixels and lane lines, thus avoiding topology problems caused by incorrect stitching.

[0027] Secondly, embodiments of this application provide a multimodal vectorized mapping apparatus, comprising: a map segmentation module for segmenting a global map to obtain local maps; the global map is composed of multimodal data; a feature extraction module for extracting multimodal features from the local maps; an instance vector prediction model for obtaining local instance vectors based on the multimodal features; the local instance vectors are data sequences of geometric semantic information of the local maps; and a vector concatenation module for obtaining a global vectorized map based on the local instance vectors; the global vectorized map is an image representing the geometric semantic information of the global map in vectorized form. Its beneficial effects are as described in the first aspect, and will not be repeated here.

[0028] In some implementations, the instance vector prediction model includes: a multimodal feature fusion module for fusing multimodal features to obtain a first fused feature; a deep supervision module for performing deep supervised learning on the first fused feature to obtain a second fused feature; and a sparse instance decoder for decoding the first fused feature and the second fused feature to obtain a local instance vector.

[0029] In some embodiments, the apparatus further includes: a pseudo-temporal feature extraction model for acquiring multimodal features of surrounding images of a local map; the surrounding images include images of the top, bottom, left, and right sides of the local map; interactive learning of the multimodal features of the local map and the multimodal features of the surrounding images using a first cross-attention mechanism to obtain optimized multimodal features; and fusing the optimized multimodal features to obtain the first fused feature.

[0030] In some embodiments, the apparatus further includes: a weighted model for performing hyperparameter learning on the multimodal features to obtain weighted weights; the weighted weights are adaptively updated according to scene changes on the local map; and a multimodal feature fusion module for merging and encoding the multimodal features according to the weighted weights to obtain the first fused feature.

[0031] In some embodiments, the apparatus further includes: a dense task learning module for determining dense features of a local map based on a first fusion feature, the dense features including semantic segmentation features and / or depth features of the local map; and a deep supervision module for interactively learning the first fusion feature and the dense features using a second cross-attention mechanism to obtain a second fusion feature.

[0032] In some implementations, the sparse instance decoder is used to perform self-learning updates of the first fused feature using a self-attention mechanism; a third cross-attention mechanism is used to interactively learn the second fused feature and the self-learned updated first fused feature to obtain the local instance vector.

[0033] In some implementations, the sparse instance decoder is also used to interactively learn the prior features and the output of the third cross-attention mechanism using a fourth cross-attention mechanism to obtain local instance vector prior features as vectorized features of the local map.

[0034] In some implementations, the multimodal features include scalarized features, and the feature extraction module includes: an image feature extractor for dividing the local map into blocks to obtain multiple block images; an image feature extraction network with multiple transformers for extracting multi-scale semantic information of each block image; and superimposing the multi-scale semantic information of each block image to obtain scalarized features of the local map.

[0035] In some implementations, the multimodal features also include vectorized features, and the feature extraction module is further used to: sample the coordinates of N points on the local map to determine the coordinates of the N points; determine the category of the coordinates of the N points; stitch together the coordinates and categories of the N points to obtain vectorized instances; and extract features from the vectorized instances to obtain the vectorized features of the local map.

[0036] In some embodiments, the apparatus further includes: a position encoder for extracting coordinate features from the vectorized instance; and a category encoder for extracting category features from the vectorized instance; the feature extraction module is also used to concatenate the coordinate features and the position features to obtain prior features.

[0037] In some implementations, there are multiple local instance vectors, and the vector concatenation module is used to: concatenate multiple local instance vectors to obtain a global instance vector; and perform sparse sampling on the point coordinates of the global instance vector to obtain a global vectorized map.

[0038] Thirdly, embodiments of this application provide a computing device, including: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, such that the computing device performs the method as described in any of the first aspects.

[0039] Fourthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method as described in any of the first aspects.

[0040] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in any of the first aspects.

[0041] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a processor, causes the processor to execute the method as described in any of the first aspects. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0044] Figure 1 is an example of the application system architecture of the multimodal vectorization mapping method proposed in the embodiments of this application;

[0045] Figure 2 is a software framework diagram of the multimodal vectorization mapping method provided in the embodiments of this application;

[0046] Figure 3a is a schematic diagram of the map segmentation module;

[0047] Figure 3b is a schematic diagram of the instance vector prediction model;

[0048] Figure 3c is a schematic diagram of the vector concatenation module;

[0049] Figure 4a is a flowchart of the multimodal vectorization mapping method provided in an embodiment of this application;

[0050] Figure 4b is a flowchart of the process of generating local instance vectors in the multimodal vectorization mapping method provided in the embodiments of this application;

[0051] Figure 5 is a schematic diagram of pseudo-temporal feature fusion in the multimodal vectorization mapping method provided in the embodiments of this application;

[0052] Figure 6 is a schematic diagram of the dynamic fusion of multimodal features in the multimodal vectorization mapping method provided in the embodiments of this application;

[0053] Figure 7 is a schematic diagram of the depth supervision module in the multimodal vectorization mapping method provided in the embodiments of this application;

[0054] Figure 8 is a schematic diagram of the sparse instance decoder in the multimodal vectorization mapping method provided in the embodiments of this application;

[0055] Figure 9 is a flowchart of the multimodal vectorization mapping method provided in Embodiment 1 of this application;

[0056] Figure 10 is a schematic diagram of vectorization processing in the method provided in Embodiment 1 of this application;

[0057] Figure 11 is a flowchart of sparse instance-level decoding provided in Embodiment 1 of this application;

[0058] Figure 12 is a flowchart of the multimodal vectorization mapping method provided in Embodiment 2 of this application;

[0059] Figure 13 is a flowchart of the sparse instance decoding process provided in Embodiment 2 of this application;

[0060] Figure 14 is a schematic diagram of a multimodal vectorization mapping device provided in an embodiment of this application;

[0061] Figure 15 is a schematic diagram of a computing device provided in an embodiment of this application;

[0062] Figure 16 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0063] Figure 17 shows a schematic diagram of a possible connection method. Detailed Implementation

[0064] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0065] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0066] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0067] In the description of the embodiments in this application, "some embodiments" are mentioned, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0068] In the description of the embodiments of this application, the terms "first, second, third, etc." or module A, module B, module C, etc. are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that, where permitted, a specific order or sequence can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0069] In the description of the embodiments of this application, the reference numerals for the steps, such as S110, S120, etc., do not necessarily indicate that the steps will be executed in this manner. Where permissible, the order of the steps can be interchanged or executed simultaneously.

[0070] Technical and scientific terms related to this application:

[0071] Artificial intelligence (AI) is a branch of computer science that attempts to understand the nature of intelligence and to produce new intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0072] A map is a graphic or image that, according to certain rules, selectively represents certain phenomena of the Earth (or other planets) in two-dimensional or multi-dimensional form and means on a plane or sphere.

[0073] Vectorization is the process of using digital image processing algorithms to identify various raster arrays on a source image as vector objects and finally save them in a certain format.

[0074] Geometric semantic information is semantic information related to geometric shapes and structures, used to describe the shape, size, orientation, and position of objects or scenes. In intelligent driving, geometric semantic information is used to identify objects such as road signs, vehicles, and pedestrians in order to plan safe driving routes.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0076] The following sections will introduce the solutions related to this application from different perspectives.

[0077] The first approach related to this application extracts semantic information from a map using a semantic segmentation method. In the preprocessing stage, image masking (Image2Mask) is used to segment the semantic information of the map, which results in relatively large noise and is easily affected by input noise. The postprocessing stage requires stitching and instantiation, and the model needs to accurately predict the forking and merging scenarios, which is logically complex and can easily lead to incorrect topological connections between instances. The output vectorized map is a semantic-level output.

[0078] The second approach related to this application uses a traffic flow clustering method to preprocess traffic flow data and then perform vector mapping using a clustering algorithm. In the traffic flow data preprocessing stage, there are many traffic flow trajectories, and the traffic flow noise is relatively large at the lane positions of variable lanes. Traffic flow noise refers to the disturbance caused by the deviation of the traffic flow trajectory. Noise can easily cause the clustering results to shift. When using the clustering algorithm to perform vector mapping, the output lane line prediction is greatly affected by noise interference and will have a lateral shift.

[0079] The third approach uses an end-to-end vectorization method based on vehicle-side surround view. It takes the surround view image as input, segments the map semantic information using a neural network, and then stitches the network outputs together for linear vectorization. However, using the surround view image as input results in significant multimodal noise interference, leading to substantial lateral shifts in the predicted lane lines and lower accuracy.

[0080] In intelligent driving technology, vectorized mapping is a map built based on vectorized data. Vectorized data is data with both size and orientation, including straight lines and curves. The vectorized mapping of a map uses straight lines and curves to describe geographic entities in the global map, including lane lines, intersection surfaces, road edges, and buildings.

[0081] In view of this, this application proposes a multimodal vectorized mapping method, which segments a global map to obtain local maps, the global map being composed of multimodal data; extracts multimodal features from the local maps; obtains local instance vectors based on the multimodal features; the local instance vectors are data sequences describing the geometric semantic information of the local maps; and concatenates the local instance vectors to obtain a global vectorized map; the global vectorized map is an image representing the geometric semantic information of the global map in vectorized form.

[0082] This application proposes a multimodal vectorization mapping method. Local instance vectors are data sequences of geometric semantic information from a local map. By obtaining local instance vectors based on the multimodal features of the local map, the geometric semantic information of the local map can be extracted without labeling the pixels on the map with entity categories. Furthermore, since the local instance vectors have a small data volume and high computational efficiency, the global vector map obtained using the local instance vectors of the local map can achieve rolling updates of the intelligent driving map, thereby helping intelligent driving vehicles better recognize their surrounding environment.

[0083] This application proposes a multimodal vectorization mapping method that can be used on various terminals, including in-vehicle systems, robots, and vehicle-mounted mobile devices.

[0084] For example, Figure 1 is an example diagram of the application system architecture for multimodal vectorized mapping provided in an embodiment of this application. As shown in Figure 1, the system architecture includes a sensor 10 and a terminal 11.

[0085] Sensor 10 includes a camera, radar, laser rangefinder, remote sensing positioning device, etc. Sensor 10 is used to collect various types of information to obtain multimodal data. Multimodal data includes point cloud data, radar remote sensing data, raw road data, raw pose data, driving trajectory, coordinate information, road direction, and directional signal sequences collected at different times and arranged chronologically. Among these, point cloud data and radar remote sensing data are scalar data; raw road data, raw pose data, driving trajectory, coordinate information, road direction, and directional signal sequences collected at different times and arranged chronologically are vectorized data.

[0086] Sensor 10 inputs multimodal data into terminal 11.

[0087] Terminal 11 includes vehicle-mounted systems, robots, and vehicle-mounted mobile devices. Terminal 11 receives multimodal data collected by sensor 10 to obtain a global map. The global map is composed of multimodal data and includes a global point cloud map, a global original road map, a global original pose map, and / or a vehicle-side surround view.

[0088] Terminal 11 executes the multimodal vectorization mapping method provided in this application embodiment, segments the global map to obtain a local map, the global map being composed of multimodal data; extracts multimodal features from the local map; obtains local instance vectors based on the multimodal features; the local instance vectors are data sequences of the geometric semantic information of the local map; obtains a global vectorized map based on the local instance vectors; the global vectorized map is an image representing the geometric semantic information of the global map in vectorized form.

[0089] The above is an introduction to the application system architecture of the multimodal vectorization mapping method provided in the embodiments of this application. The software architecture of the multimodal vectorization mapping method is described below.

[0090] Figure 2 is a software architecture diagram provided in an embodiment of this application. As shown in Figure 2, it includes a map segmentation module 11, a feature extraction module 12, an instance vector prediction model 13, and a vector concatenation module 14.

[0091] The map segmentation module 11 segments the global map to obtain local maps. The global map is composed of multimodal data.

[0092] For example, Figure 3a is a schematic diagram of the map segmentation module. As shown in Figure 3a, the global map consists of multimodal data, including point cloud data and / or raw road data. The segmented local maps include local point cloud maps and / or local raw road maps, etc. It can be understood that the local point cloud map and the local raw road map are multimodal representations of the same local map. There are multiple segmented local maps.

[0093] In some possible implementations, before segmenting the global map, the vehicle acquires multimodal data by receiving data collected by sensors during intelligent driving, and generates a global map based on the multimodal data.

[0094] Feature extraction module 12 extracts multimodal features from the local map.

[0095] In some possible implementations, the multimodal features include scalarized features, and the feature extraction module includes: an image feature extractor that divides the local map into blocks to obtain multiple block images; an image feature extraction network with multiple transformers that extracts multi-scale semantic information of each block image; and superimposing the multi-scale semantic information of each block image to obtain scalarized features of the local map.

[0096] The image feature extractor includes M swin transformer blocks, a partitioning module, a linear embedding module, and a block fusion module. The partitioning module divides the local map into blocks to obtain multiple patch images. The linear embedding module projects the dimension of each patch image to a higher dimension. The patch fusion module halves the height and width of the patch images. The swin transformer block is a transformer-based image feature extraction network. The M swin transformer blocks are stacked to restrict the transformer to a single window. By moving the window, the patch images of different windows are linked together to output local image features, which are scalarized features.

[0097] During CNN learning and training, the image is first divided into multiple small blocks. The image encoder (or filter or feature detector) only looks at one block of the image at a time. This small block is called a patch. The patch represents the geometric features of the image region.

[0098] In some possible implementations, the multimodal features also include vectorized features. The feature extraction module 12 determines the coordinates of N points by sampling the coordinates of N points on the local map; determines the category of the coordinates of the N points; stitches the coordinates and categories of the N points together to obtain vectorized instances; and extracts features from the vectorized instances to obtain the vectorized features of the local map.

[0099] In some possible implementations, the architecture also includes a position encoder and a category encoder, wherein the position encoder extracts coordinate features from the vectorized instance; the category encoder extracts category features from the vectorized instance; and the feature extraction module 12 concatenates the coordinate features and the position features to obtain prior features.

[0100] Prior features are features extracted from prior information. Prior information is a structured set of information formed by superimposing geometric, semantic, attribute, and category information. For example, attribute information includes right turn and straight ahead; category information includes lane lines and roundabouts.

[0101] Instance vector prediction model 13 obtains local instance vectors based on multimodal features; local instance vectors are data sequences of geometric semantic information of local maps.

[0102] In some possible implementations, the instance vector prediction model 13 fuses multimodal features to obtain a first fused feature; performs deep supervised learning based on the first fused feature to obtain a second fused feature; and decodes based on the first fused feature and the second fused feature to obtain a local instance vector.

[0103] For example, Figure 3b is a schematic diagram of the structure of the instance vector prediction model. As shown in Figure 3b, the instance vector prediction model 13 includes a multimodal feature fusion module 131, a deep supervision module 132, and a sparse instance decoding module 133.

[0104] The multimodal feature fusion module 131 is used to encode and fuse local multimodal features to obtain fused features, which can be denoted as the first fused feature.

[0105] In some possible implementations, the architecture also includes a pseudo-temporal feature extraction model, which acquires multimodal features of the surrounding images of the local map; the surrounding images include the top, bottom, left and right images of the local map; the multimodal feature fusion module 131 uses a first cross-attention mechanism to interactively learn the multimodal features of the local map and the multimodal features of the surrounding images to obtain optimized multimodal features; the optimized multimodal features are fused to obtain the first fused feature.

[0106] In some possible implementations, the architecture also includes a weighted model that performs hyperparameter learning on the multimodal features to obtain weights; the weights are adaptively updated according to scene changes on the local map; and the multimodal features are merged and encoded according to the weights to obtain the first fused feature.

[0107] In some possible implementations, the architecture further includes a dense task learning module that performs dense task learning on the local map to obtain dense features; the dense features include semantic segmentation features and / or depth features of the local map; the deep supervision module 132 uses a second cross-attention mechanism to interactively learn the first fused features and the dense features to obtain the second fused features.

[0108] In some possible implementations, the first fused feature and the dense feature can interact using a second cross-attention mechanism, the weights can be determined using a Softmax classifier, and the enhanced feature can be obtained after calculation by MatMul. The first fused feature and the enhanced feature can be merged together to obtain the local fused feature, which can be denoted as the second fused feature.

[0109] Here, dense features are intermediate embedding vectors that precede the classification results output for dense tasks.

[0110] Dense tasks involve classifying pixels, including various visual tasks such as semantic segmentation and / or deep supervision. Semantic segmentation classifies every pixel in a local scene of an image. Deep supervision adds an auxiliary classifier, Softmax, as a branch of the network in the middle hidden layers to supervise the backbone network, addressing issues such as vanishing gradients and slow convergence during training in deep neural networks.

[0111] The deep supervision module 132 learns the interaction between the first fused features and the features of the dense task, which improves the model's representation ability and allows the model's cross-attention mechanism to focus more on the effective region, thus accelerating convergence and model representation.

[0112] The sparse instance decoder 133 decodes the first fusion feature and the second fusion feature to obtain a local instance vector prediction. The sparse instance decoder includes a self-attention mechanism and a third cross-attention mechanism. The input of the sparse instance decoder includes the first fusion feature and the second fusion feature. The first fusion feature is updated through self-learning using the self-attention mechanism. The third cross-attention mechanism is used to interactively learn the second fusion feature and the self-learned updated first fusion feature to obtain a local instance vector.

[0113] In some possible implementations, the input to the sparse instance decoder also includes prior features (Prior Embedding), and the sparse instance decoder also includes a fourth cross-attention mechanism, which can be used to interactively learn the prior features and the output of the third cross-attention mechanism to obtain local instance vectors.

[0114] Prior features and features from other modalities are enhanced using a cross-attention mechanism to strengthen the query (Q), thereby enabling better acquisition of multimodal features and prior features.

[0115] Vector stitching module 14 is used to stitch together local instance vectors to obtain a global vectorized map.

[0116] In some possible implementations, there are multiple local instance vectors. The vector splicing module 14 splices multiple local instance vectors to obtain a global instance vector. The point coordinates of the global instance vector are sparsified and sampled to obtain a global vectorized map.

[0117] A global vectorized map is an image representing the geometric semantic information of a global map in vector form.

[0118] For example, Figure 3c is a schematic diagram of the vector concatenation module. As shown in Figure 3c, multiple local instance vectors are concatenated, and the point coordinates of the concatenated global instance vector are sparsely sampled to obtain a global vectorized map.

[0119] Therefore, the multimodal vectorized mapping method provided in this application uses an instance vector prediction model to perform deep supervised learning on local fusion features, and uses Softmax classification to automatically classify each pixel of the local map without labeling pixels; it predicts local instance vectors through the instance vector prediction model, and performs vectorized instance splicing based on the geometric semantic information of the local instance vectors, without the need for complex post-processing, thus avoiding topological problems caused by incorrectly attaching vectorized instances.

[0120] The multimodal vectorization mapping method provided in this application is based on the process of constructing a global vectorized map by stitching together local instance vectors from multimodal data collected by sensors during intelligent driving received by the vehicle. It features small data volume, high computational efficiency, and strong real-time performance, and can achieve rolling updates. The multimodal vectorization mapping method provided in this application will be described below based on the above description.

[0121] Figure 4a is a flowchart of the multimodal vectorized mapping method provided in an embodiment of this application. Some or all of the steps of the multimodal vectorized mapping method proposed in this application can be referred to the relevant descriptions in Figures 1-3 above. It is understood that this method can be executed by any device or terminal with computing power. As shown in Figure 4a, the method includes: S11, segmenting the global map to obtain a local map; S12, extracting multimodal features from the local map; S13, obtaining local instance vectors based on the multimodal features; the local instance vectors are data sequences describing the geometric semantic information of the local map; S14, obtaining a global vectorized map based on the local instance vectors; the global vectorized map is an image representing the geometric semantic information of the global map in vectorized form.

[0122] The above steps will be further described below with reference to Figure 4a and specific implementation methods.

[0123] S11, the global map is segmented to obtain local maps. The global map is composed of multimodal data.

[0124] In some possible implementations, before segmenting the global map, the vehicle acquires multimodal data by receiving data collected by sensors during intelligent driving, and generates a global map based on the multimodal data.

[0125] Sensors include cameras, radar, laser rangefinders, and remote sensing positioning devices; sensors collect various types of information and obtain multimodal data.

[0126] Multimodal data includes point cloud data, radar remote sensing data, raw road data, raw pose data, driving trajectory, coordinate information, road direction, and directional signal sequences collected at different times and arranged in time.

[0127] Point cloud data and radar remote sensing data are scalar data; raw road data, raw pose data, driving trajectory, coordinate information, road direction, and directional signal sequences collected at different times and arranged in time are vectorized data.

[0128] The global map consists of multimodal data, including a global point cloud map, a global original road map, a global original pose map, and / or a vehicle-side surround view.

[0129] The segmented local maps are multiple, and each local map consists of multimodal data, including local point cloud maps, local original road maps, local original pose maps, and / or vehicle-side surround views, etc.

[0130] Global maps can be categorized into global scalar maps and / or global vector maps based on the attributes of the data.

[0131] Local maps can be categorized into local scalar maps and / or local vector maps based on the attributes of the data.

[0132] S12, extract local multimodal features from the local map.

[0133] In some possible implementations, the local multimodal features include scalarized features. The local map is input into an image feature extractor, which divides the local map into blocks to obtain multiple block images. Multi-scale semantic information of each block image is extracted using an image feature extraction network with multiple transformers. The multi-scale semantic information of each block image is superimposed to obtain the scalarized features of the local map.

[0134] The image feature extractor includes M swin transformer blocks, a partitioning module, a linear embedding module, and a block fusion module. The partitioning module divides the local map into blocks to obtain multiple patch images. The linear embedding module projects the dimension of each patch image to a higher dimension. The patch fusion module halves the height and width of the patch images. The swin transformer block is a transformer-based image feature extraction network. The M swin transformer blocks are stacked to restrict the transformer to a single window. By moving the window, the patch images of different windows are linked together to output local image features, which are scalarized features.

[0135] Geometric information of the corresponding image on the local map can be obtained based on local image features.

[0136] In some possible implementations, local multimodal features include vectorized features, which can acquire multiple vectorized data points of a local map, sample N points in each vectorized data point to determine the coordinates of the N points; determine the category of the coordinates of the N points; stitch together the coordinates and categories of the N points to obtain vectorized instances; and extract features from the vectorized instances to obtain the vectorized features of the local map.

[0137] For example, the coordinates of N points on the local original road map are sampled and normalized, and the categories of the N points on the local original road map are one-hot encoded to obtain the category code and coordinate code of the N sampled points on the local original road map. The coordinate code and category code are combined and concatenated to obtain a vectorized instance of the local original road map. The vectorized instance is input into the RP (row lane) encoder and the positional encoding (PE) to extract category features and position features from the vectorized instance of the local original road map to obtain the vectorized features of the local original road map.

[0138] In some possible implementations, the local map includes a local original pose map. The coordinates of N points on the local original pose map are sampled and normalized, and the categories of the N points on the local original pose map are one-hot encoded to obtain the category codes and coordinate codes of the N sampled points on the local original pose map. The coordinate codes and category codes are combined and concatenated to obtain a vectorized instance of the local original pose map. The vectorized instance is input into the RP (row lane) encoder and the positional encoding (PE) to extract category features and position features from the local original pose map to obtain the vectorized instance features of the local original pose map.

[0139] Among them, the RL (row lane) encoder and the RP (row pose) encoder both belong to the category encoder.

[0140] S13, obtain local instance vectors based on local multimodal features; local instance vectors are data sequences of geometric semantic information of the local map.

[0141] In some possible implementations, the local instance vector is obtained through the following steps S131-S133. As shown in Figure 4b, the details are explained below.

[0142] S131, the local multimodal features are encoded and fused to obtain the first fused feature.

[0143] In some possible implementations, multimodal features of the surrounding images of the local map can be obtained through a pseudo-temporal feature extraction model; a first cross-attention mechanism is used to interactively learn and fuse the multimodal features of the local map and the multimodal features of the surrounding images of the local map to obtain a first fused feature.

[0144] Specifically, the pseudo-temporal feature extraction model uses local multimodal features as the central feature, which is Q (query), and the surrounding moment features as K and V, employing a first cross-attention mechanism for interactive learning. The moment features of the surrounding images include vectorized features of the moments of the images above, below, to the left, and to the right of the central feature. This cross-attention mechanism can be referred to as the first cross-attention mechanism.

[0145] For example, Figure 5 is a schematic diagram of pseudo-temporal feature fusion in the multimodal vectorization mapping method provided in the embodiments of this application. As shown in Figure 5, the map is first divided into multiple local maps, with the local map as the central feature, the central feature as Q, and the moment features of the surrounding images as K and V. The first cross-attention mechanism is used for interactive learning to obtain optimized multimodal features.

[0146] In some possible implementations, the local multimodal features include scalarized image features, with the scalarized image features of the local map as the center feature (Q), and the moment features of the surrounding images (K and V). An interactive learning process using a first cross-attention mechanism is employed to obtain optimized multimodal features. The moment features of the surrounding images include scalarized features of the moments of the images above, below, to the left, and to the right of the center feature.

[0147] In some possible implementations, the local multimodal features include vector image features, with the vectorized features of the local map as the center feature, based on pseudo-temporal image features, the center feature as Q, and the moment features of the surrounding images as K and V, and interactive learning is performed using a first cross-attention mechanism to obtain optimized multimodal features.

[0148] By fusing pseudo-temporal multimodal features, the reference scope for prediction can be expanded, the model's ability to extract semantic and geometric features can be strengthened, the network's recall ability can be improved, and the ability to reproduce roads can be enhanced.

[0149] In some possible implementations, local multimodal features can be input into a weighted model, which performs hyperparameter learning on the multimodal features to obtain weights; the weights are dynamically updated adaptively based on scene changes on the local map; and the multimodal features are merged and encoded based on the weights.

[0150] In some possible implementations, pseudo-temporally optimized local multimodal features can be input into a weighted model, which then performs hyperparameter learning on the optimized multimodal features to obtain weights. The weights are dynamically updated adaptively based on scene changes on the local map, and the optimized multimodal features are merged and encoded according to the weights.

[0151] For example, Figure 6 is a schematic diagram of the dynamic fusion of multimodal features in the multimodal vectorization mapping method provided in the embodiments of this application. As shown in Figure 6, the method includes the following steps:

[0152] S1311, Obtain the multimodal features of the global multimodal data.

[0153] For example, the multimodal data includes point cloud maps, original road maps, and original poses; embedding features of local point cloud maps (PC), local original road maps (RL), and local original poses (RP) are extracted.

[0154] S1312 merges multimodal features.

[0155] For example, the embedding features of the local point cloud map PC, the embedding features of the local original road map RL, and the embedding features of the local original pose RP are used.

[0156] S1313, the merged features are subjected to N times of global average pooling and multi-channel attention mechanism to learn the weighted weights (add) between features.

[0157] S1314, perform hyperparameter learning, and determine the dynamic weighting weight of each modality feature in the multimodal features through SoftMax classification.

[0158] For example, the multimodal features include PC embedding features, RP embedding features, and RP embedding features; the dynamic weighting of the multimodal features includes the weighting of PC embedding features, the weighting of RP embedding features, and the weighting of RP embedding features.

[0159] In some possible implementations, the first fusion feature can be obtained using the following formula:

[0160] fusion in formula (1) feat The first fusion feature is weight1, weight2, ..., weightn, which are dynamically learnable noise-resistant weights for different scenarios. This involves merging features from different modalities.

[0161] Formula (1) means that the weighted weights between features are learned by using global features + local features. Features of different modalities are complementary. For noisy data, smaller weights can be learned adaptively to reduce the impact of noise.

[0162] S132, the first fusion feature and the dense feature are interactively learned to obtain the second fusion feature; the dense feature includes semantic segmentation features and / or depth features of the local map.

[0163] In some possible implementations, dense features of the local map can be determined, and a second cross-attention mechanism can be used to interactively learn the first fused features and the dense features.

[0164] For example, Figure 7 is a schematic diagram of the deep supervision module in the multimodal vectorization mapping method provided in this application embodiment. As shown in Figure 7, the first fusion feature is Q, and the dense features are K and V. Interactive learning is performed using a second cross-attention mechanism. The weights W are determined by a Softmax classifier. The dense features are calculated using MatMul to obtain the enhanced features. The first fusion feature and the enhanced features are merged to obtain the second fusion feature. Specifically, as shown in the formula:

[0165] Enhanced in formula (2) feat To enhance features, Fusion feat As the first fusion feature, Seg feat Dense characteristics; Fusedfeat The second fusion feature is represented by formula (2), which indicates that the dense feature and the first fusion feature are interactively learned using the second cross-attention mechanism. The weights are determined by the Softmax classifier, and the enhanced feature is obtained after calculation using the MatMul matrix. feat =Cat[Seg feat Enhanced feat (3)

[0166] Formula (3) indicates that the dense feature and the enhanced feature are combined (concat) to obtain the second fused feature.

[0167] The deep supervision module seg GT learns K and V by minimizing the object detection loss function loss. It calculates the loss based on the loss value between the predicted instance vectorization vector and the ground truth box / line, and adjusts the weights of dense features through the backpropagation algorithm.

[0168] S133, a sparse instance decoder is used to decode the first fusion feature and the second fusion feature to output a local instance vector prediction. The sparse instance decoder includes a self-attention mechanism and a third cross-attention mechanism. The input of the sparse instance decoder includes the first fusion feature and the second fusion feature. The first fusion feature is updated through self-learning using the self-attention mechanism. The third cross-attention mechanism is used to interactively learn the second fusion feature and the self-learned updated first fusion feature to obtain the local instance vector.

[0169] For example, Figure 8 is a schematic diagram of a sparse instance decoder in the multimodal vectorization mapping method provided in this application embodiment. As shown in Figure 8, the sparse instance decoder decodes the input to obtain the extraction and understanding of the geometric semantic information of the local map, and generates local instance vectors.

[0170] The input to the sparse instance-level decoder includes a learnable query (Q) and a second fused feature. feat ) and prior features (prior embedding).

[0171] The learnable query Q is a feature vector used for instance detection. In this embodiment, the initialized Q can be a first fusion feature with a scale of N×M×dim.

[0172] In some possible implementations, the input to the sparse instance decoder also includes prior features, which are vectorized instance features of the local map. The sparse instance decoder also includes a fourth cross-attention mechanism, which can be used to interactively learn the prior features and the output of the third cross-attention mechanism.

[0173] S14, derives the global vectorized map from the local instance vectors. The global vectorized map is a global map displayed using lines and curves based on the geometric semantic information of the local maps.

[0174] It is understandable that the global map can be divided into multiple local maps, and multiple local maps correspond to multiple local instance vectors.

[0175] In some possible implementations, there are multiple local instance vectors, and multiple local instance vectors are concatenated to obtain a global instance vector; the point coordinates of the global instance vector are sparsely sampled to obtain a global vectorized map.

[0176] The multimodal vectorization mapping method provided in this application can be used for geometric mapping of intelligent driving maps and remote sensing / aerial images. It includes the extraction of geometric attributes of vectorized instances such as lane lines, road boundaries, intersection surfaces, pedestrian crossings, and buildings, as well as the extraction of city-level geometric semantic information, which significantly improves the mapping quality of intelligent driving maps.

[0177] Example 1

[0178] The multimodal vectorization mapping method provided in Embodiment 1 of this application is applied to the scenario of automatic extraction of vectorized semantics from intelligent driving maps.

[0179] Figure 9 is a flowchart of the multimodal vectorization mapping method provided in Embodiment 1 of this application. As shown in Figure 9, the global map is segmented to obtain local maps. The global map is composed of multimodal data, including scalarized data and / or vectorized data. The processing of the scalarized data includes the following steps:

[0180] S911 performs a fixed H×W sliding window segmentation on the scalarized global map to obtain scalarized data for multiple local maps. The scalarized data of the local maps comes from the input of the scalar mode.

[0181] S921, extract scalarized features from the local map.

[0182] For example, a feature extractor with a swin transformer network structure can be used to extract scalar features from scalar data of the Local Map. The feature extractor's network structure consists of M stacked swin transformer blocks; where patch partitioning divides the input H×W Local Map image into 16 blocks to obtain 16 block images; linear embedding projects the dimension of each block image from 16 to C; patch merging halves the height and width of the image; the swin transformer block is a transformer-based image feature extraction network used to restrict the transformer to a single window and connect images from different windows by moving the window; thus, the feature extraction capability of the image is enhanced while reducing the complexity of the model.

[0183] The scalarized data of the local map is processed through M swin transformer blocks. The feature map output by each swin transformer block has a resolution half that of the previous stage. The network structure can output feature maps of M stages. A feature pyramid is obtained based on the feature maps of the M stages. The feature pyramid includes multi-scale semantic information. The feature pyramid is upsampled, and the multi-scale semantic information is superimposed after the height and width of the image are reduced to a fixed scale of 1 / 2M to obtain the scalarized features of the scalarized data, which are then used by downstream modules.

[0184] The processing of vectorized data includes the following steps:

[0185] S912 performs a fixed H×W sliding window segmentation on the global map to obtain vectorized data of the local map. The vectorized data of the local map includes lane lines, road boundaries, intersection surfaces, and the size and direction of pedestrian crossings described by straight lines and curves.

[0186] S922, determine the coordinates and categories of N sampling points on each vectorized data, encode and concatenate the coordinates and categories of the N sampling points to obtain a vectorized instance.

[0187] For example, Figure 10 is a schematic diagram of vectorized feature extraction provided in Embodiment 1 of this application. As shown in Figure 10, for each input vectorized data (PolyLine), the coordinate information of each vectorized data is normalized by sampling the coordinates of N fixed points, and the coordinate codes of the N points corresponding to each vectorized data are obtained as pos=[(x1,y1),(x2,y2),…,(xn,yn)]. According to the category of each vectorized data, a one-hot category code one_hot=[0,0,1,0] can be generated. The coordinate codes and category codes of each vectorized data are combined and concatenated to obtain the vectorized instance Instance of the Local Map: Instance=concatenate(pos,one_hot).

[0188] The categories of vectorized data include lane line attributes such as straight and right turn.

[0189] S923 extracts features from vectorized instances of a local map to obtain vectorized features of the local map.

[0190] In some possible implementations, continuing as shown in Figure 10, a rowlane encoder can be used to extract class features from the vectorized instances of N points to obtain instance vector embedding features. A positional encoder (PE) can be used to extract coordinate features from the vectorized instances of N points to obtain coordinate embedding features (PE embedding). The instance vector embedding features and the coordinate embedding features are concatenated to obtain prior features (Prior embedding). The prior features serve as vectorized features of the local map. Subsequently, a fourth cross-attention mechanism is used to interactively learn features from other modalities to strengthen the prior query, thereby enabling the query to better acquire multimodal features and prior information.

[0191] S93 fuses scalarized features and vectorized features to obtain local fused features, performs deep supervised learning and sparse instantiation decoding on the local fused features, and outputs local instance vectors.

[0192] The specific implementation method for fusing scalarized features and vectorized features to obtain local fused features can refer to some or all of the implementation methods in step S131. It will not be repeated here.

[0193] The specific implementation of deep supervised learning based on local fusion features can be part or all of the implementation of step S132, which will not be described in detail here.

[0194] Figure 11 is a flowchart of the sparse instantiation decoding process provided in Embodiment 1 of this application. As shown in Figure 11, in Embodiment 1 of this application, the sparse instantiation decoding of local fusion features includes the following steps:

[0195] The input is fed into a sparse instance-level decoder for decoding. The input to the sparse instance-level decoder includes a learnable query, prior features, and a second fused feature.

[0196] The learnable query is a feature vector used for lane detection, with a scale of N×M×dim, where N is the total number of lane instances and M is the number of points on each lane. The query is initialized through a fully connected layer.

[0197] Specifically, for each query, it is first initialized as a random vector, and then mapped to a vector with the same number of targets through a fully connected layer. The weights of the fully connected layer are randomly initialized.

[0198] During training, the instance vector prediction model learns the query by minimizing the object detection loss function. The model calculates the loss based on the loss value between the predicted box / line and the ground truth box / line, and adjusts the weight of the query through the backpropagation algorithm to better match the target.

[0199] The prior features are derived from each vectorized data point after vectorization encoding. By sampling the coordinates of N keypoints for each vectorized data point, its coordinate encoding is obtained: pos = [(x1,y1),(x2,y2),…,(xn,yn)]. Based on its category, a one-hot category encoding one_hot = [0,0,1,0] can be generated. Finally, the coordinate encoding and category encoding are combined and concatenated to obtain the instance: Instance = concatenate(pos,one_hot). The feature generated by the category encoder after encoding the instance is the category feature, and the feature generated by the position encoder after encoding the instance is the position feature. Finally, the category feature and the position feature are superimposed to form the prior features, which are used as one input to the second cross-attention mechanism. This can better represent its position information and category information without the need to label the pixels on the map with entity categories.

[0200] The fused features are scalarized features from the Local Map extracted by the image feature extractor.

[0201] In some possible implementations, information from the local map can be fed into a feature extractor to extract visual features, and the resulting image features can be used as one input to the third cross-attention mechanism.

[0202] In some possible implementations, the first fused feature can be used as the query. After the initialized query enters the sparse instance-level decoder, it undergoes a self-attention process to update the query's features.

[0203] Next, a third cross-attention mechanism is used to perform feature interaction on the self-learned updated query and the second fused feature.

[0204] In this process, the initialized query is used as the query, and the second fusion feature output by the multimodal fusion model 131 is used as the key and value for feature interaction learning. This process promotes information exchange between and within instances, thereby further improving the learning performance of the instance vector prediction model.

[0205] After the first interaction, the fourth cross-attention mechanism is used to perform another feature interaction on the updated query and prior features, and output a local instance vector.

[0206] In this process, prior features, acting as both key and value, interact with the updated query output by the third cross-attention mechanism. This process further improves the model's accuracy and robustness, enabling it to better adapt to various real-world application scenarios.

[0207] S94: Concatenate the local instance vectors corresponding to multiple local maps to obtain a global instance vector, and resample the point coordinates of the global instance vector to obtain a global vectorized map.

[0208] Example 2

[0209] The multimodal vectorization mapping method provided in Embodiment 2 of this application differs from Embodiment 1 in that the vehicle end does not have prior RoadCode feature input, so the process can be appropriately simplified and vectorization processing is not required.

[0210] Figure 12 is a flowchart of the multimodal vectorization mapping method provided in Embodiment 2 of this application. As shown in Figure 12, the global map is segmented to obtain local maps. The global map is composed of multimodal data, which includes various types of standardized data. The processing of the standardized data includes the following steps:

[0211] S1201 performs a fixed H×W sliding window segmentation on the scalarized global map to obtain scalarized data of multiple local maps. The scalarized data of the local maps comes from the input of the scalar mode.

[0212] S1202, extract the scalarized features of each local map.

[0213] A feature extractor with a Swing transformer network structure is used to extract scalar features from the scalar data of the Local Map. For specific implementation details, please refer to step S821 in Example 1, which will not be repeated here.

[0214] S1203 fuses scalarized features to obtain local fused features, performs deep supervised learning and sparse instance decoder decoding on the local fused features, and outputs local instance vectors.

[0215] The fusion of scalarized features to obtain local fused features involves two processes: pseudo-temporal feature fusion and dynamic weighted fusion of multimodal features. The implementation steps can be as described in step S131. The implementation methods in this application will not be repeated here.

[0216] Deep supervised learning is performed on the local fusion features. For specific implementation methods, please refer to step S132, which will not be repeated here.

[0217] In the scenario of Embodiment 2 of this application, the vehicle does not have the RoadCode prior feature PE input, so there is no need to perform a second cross-attention learning. The local fusion features are sparsely instantiated and decoded.

[0218] For example, Figure 13 is a flowchart of the sparse instance-level decoding process provided in Embodiment 2 of this application. As shown in Figure 13, the fused features can be scalarized features extracted from the visual backbone.

[0219] In some possible implementations, information from the local map is fed into a feature extractor to extract visual features, and the second fused feature output after fusion by the multimodal fusion model 131 is used as one input to the cross-attention mechanism.

[0220] In some possible implementations, the first fused feature is used as the query. After the query enters the sparse instance-level decoder, it undergoes a self-learning self-attention mechanism process to update the query's features.

[0221] Next, a third cross-attention mechanism is used to perform feature interaction on the new query learned by self-learning and the second fusion feature.

[0222] In this process, the self-learned updated query is used as the query, and the second fusion feature is used as the key and value to interact with the query. This promotes information exchange between and within instances, thereby further improving the learning performance of the instance vector prediction model.

[0223] S1204: Concatenate the local instance vectors corresponding to multiple local maps to obtain a global instance vector, and resample the point coordinates of the global instance vector to obtain a global vectorized map.

[0224] The above is an introduction to the multimodal vectorization mapping method provided by the embodiments of this application. It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, in some possible implementations, each step in the above embodiments may be selectively executed according to the actual situation, and may be partially or fully executed, without limitation here. Furthermore, all or part of any feature of any of the above embodiments may be freely and arbitrarily combined without contradiction; the combined technical solution is also within the scope of this application.

[0225] Next, based on the above, the apparatus for multimodal vectorization mapping provided in the embodiments of this application will be described. For details regarding the concepts, formulas, etc., involved in the following content, please refer to the above text.

[0226] Figure 14 illustrates a multimodal vectorization mapping apparatus provided in an embodiment of this application. As shown in Figure 14, the multimodal vectorization mapping apparatus 110 includes a map segmentation module 11, a feature extraction module 12, an instance vector prediction model 13, and a vector stitching module 14.

[0227] The map segmentation module 11 is used to segment the global map to obtain local maps; the global map is composed of multimodal data.

[0228] Feature extraction module 12 is used to extract multimodal features of the local map.

[0229] Instance vector prediction model 13 is used to obtain local instance vectors based on the multimodal features; the local instance vectors are data sequences of geometric semantic information of the local map.

[0230] The vector stitching module 14 is used to obtain a global vectorized map based on local instance vectors; the global vectorized map is an image representing the geometric semantic information of the global map in vectorized form.

[0231] In some possible implementations, the multimodal feature fusion module includes: the apparatus further includes: a pseudo-temporal feature extraction model for acquiring multimodal features of surrounding images of a local map; the surrounding images include images of the top, bottom, left, and right sides of the local map; the multimodal feature fusion module is used to interactively learn the multimodal features of the local map and the multimodal features of the surrounding images using a first cross-attention mechanism to obtain optimized multimodal features; and to fuse the optimized multimodal features to obtain the first fused feature.

[0232] In some possible implementations, the multimodal features include optimized multimodal features, and the apparatus further includes: a weighted model for performing hyperparameter learning on the multimodal features to obtain weighted weights; the weighted weights are adaptively updated according to scene changes on the local map; and a multimodal feature fusion module for merging and encoding the multimodal features according to the weighted weights to obtain the first fused feature.

[0233] In some possible implementations, the apparatus further includes: a dense task learning module for determining dense features of a local map based on a first fusion feature, the dense features including semantic segmentation features and / or depth features of the local map; and a deep supervision module for interactively learning the first fusion feature and the dense features using a second cross-attention mechanism to obtain a second fusion feature.

[0234] In some possible implementations, the sparse instance decoder includes a self-attention mechanism and a third cross-attention mechanism; the input of the sparse instance decoder includes a first fusion feature and a second fusion feature; the sparse instance decoder is used to perform self-learning updates on the first fusion feature using the self-attention mechanism; the second fusion feature and the self-learned updated first fusion feature are interactively learned using the third cross-attention mechanism to obtain the local instance vector.

[0235] In some possible implementations, the input to the sparse instance decoder also includes prior features, which are vectorized features of the local map; the sparse instance decoder also includes a fourth cross-attention mechanism; the sparse instance decoder is also used to interactively learn the prior features and the output of the third cross-attention mechanism using the fourth cross-attention mechanism to obtain local instance vectors.

[0236] In some possible implementations, the multimodal features include scalarized features, and the feature extraction module includes: an image feature extractor for dividing the local map into blocks to obtain multiple block images; extracting multi-scale semantic information of each block image using an image feature extraction network with multiple transformers; and superimposing the multi-scale semantic information of each block image to obtain the scalarized features of the local map.

[0237] In some possible implementations, the multimodal features also include vectorized features, and the feature extraction module is further used to: sample the coordinates of N points on the local map to determine the coordinates of the N points; determine the category of the coordinates of the N points; stitch together the coordinates and categories of the N points to obtain vectorized instances; and extract features from the vectorized instances to obtain the vectorized features of the local map.

[0238] In some possible implementations, the apparatus further includes: a position encoder for extracting coordinate features from the vectorized instance; and a category encoder for extracting category features from the vectorized instance; the feature extraction module is also used to concatenate the coordinate features and the position features to obtain prior features.

[0239] In some possible implementations, there are multiple local instance vectors, and the vector concatenation module is used to: concatenate multiple local instance vectors to obtain a global instance vector; and perform sparse sampling on the point coordinates of the global instance vector to obtain a global vectorized map.

[0240] As an example of a software functional unit, the map segmentation module 11 may include code running on computing instances. These computing instances may include at least one of physical hosts (computing devices), virtual machines, and containers. Furthermore, the aforementioned computing instances may be one or more. For example, the map segmentation module 11 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0241] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0242] As an example of a hardware functional unit, the map segmentation module 11 may include at least one computing device, such as a computing device. Alternatively, the map segmentation module 11 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0243] The map segmentation module 11 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the map segmentation module 11 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the map segmentation module 11 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as computing devices, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0244] It should be noted that, in other embodiments, the steps implemented by the map segmentation module 11, feature extraction module 12, instance vector prediction model 13, and vector stitching module 14 shown in FIG14 can be specified as needed. The map segmentation module 11, feature extraction module 12, instance vector prediction model 13, and vector stitching module 14 respectively implement different steps in the method provided in FIG3 to realize all the functions of the multimodal vectorization mapping device 110.

[0245] Figure 15 is a schematic diagram of a computing device also provided in this application. As shown in Figure 15, the computing device 1000 includes: a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a computing device or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0246] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 15, but this does not imply that there is only one bus or one type of bus. Bus 1004 can include pathways for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0247] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0248] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0249] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the map segmentation module 11 shown in FIG3, thereby implementing all or part of the steps of the method in the above embodiments. That is, the memory 1006 stores instructions for executing all or part of the steps of the method in the above embodiments.

[0250] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0251] This application also provides a computing device cluster. The computing device cluster includes at least one computing device 1000. This computing device can be a central computing device, an edge computing device, or a local computing device in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0252] As shown in Figure 16, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for performing all or part of the steps in the methods of the above embodiments.

[0253] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for performing all or part of the steps in the methods of the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for performing all or part of the steps in the methods of the above embodiments.

[0254] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, which are used to execute some functions of the multimodal vectorization mapping 110. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules among the map segmentation module 11, feature extraction module 12, instance vector prediction model 13, and vector stitching module 14 shown in Figure 14 above.

[0255] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 17 illustrates one possible connection method. As shown in Figure 17, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the memory 1006 in computing device 1000A stores instructions for the functions of the map segmentation module 11 and the performance data acquisition module 122. Simultaneously, the memory 1006 in computing device 1000B stores instructions for executing the functions of the data analysis module 131 and the visualization map generation module 132.

[0256] It should be understood that the functions of computing device 1000A shown in Figure 17 can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0257] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0258] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0259] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A multimodal vectorization mapping method, characterized in that, The method includes: A local map is obtained by segmenting the global map, wherein the global map is composed of multimodal data; Extract the multimodal features of the local map; Local instance vectors are obtained based on the multimodal features; the local instance vectors are data sequences of the geometric semantic information of the local map; A global vectorized map is obtained based on the local instance vectors; the global vectorized map is an image representing the geometric semantic information of the global map in vectorized form.

2. The method according to claim 1, characterized in that, The step of obtaining the local instance vector based on the multimodal features includes: The multimodal features are fused to obtain the first fused feature; Deep supervised learning is performed on the first fused feature to obtain the second fused feature; The first fusion feature and the second fusion feature are decoded to obtain the local instance vector.

3. The method according to claim 2, characterized in that, The method further includes: Obtain multimodal features of the surrounding images of the local map; the surrounding images include the images above, below, to the left, and to the right of the local map; The process of fusing the multimodal features to obtain the first fused feature includes: The first cross-attention mechanism is used to interactively learn the multimodal features of the local map and the multimodal features of the surrounding image to obtain optimized multimodal features; The optimized multimodal features are fused to obtain the first fused feature.

4. The method according to claim 2, characterized in that, The method further includes: The multimodal features are subjected to hyperparameter learning to obtain weighted weights; the weighted weights are adaptively updated according to scene changes on the local map. The process of fusing the multimodal features to obtain the first fused feature includes: The multimodal features are merged and encoded according to the weighted weights to obtain the first fused feature.

5. The method according to any one of claims 2-4, characterized in that, The method further includes: Dense task learning is performed on the local map to obtain dense features; the dense features include semantic segmentation features and / or depth features of the local map; The step of performing deep supervised learning based on the first fusion feature to obtain the second fusion feature includes: The second fusion feature is obtained by interactively learning the first fused feature and the dense feature using a second cross-attention mechanism.

6. The method according to any one of claims 2-5, characterized in that, The step of decoding based on the first fusion feature and the second fusion feature to obtain the local instance vector includes: The first fused feature is updated through self-learning using the self-attention mechanism. The second fused feature and the self-learning updated first fused feature are interactively learned using a third cross-attention mechanism to obtain the local instance vector.

7. The method according to claim 6, characterized in that, After the second fusion feature and the self-learned updated first fusion feature are interactively learned using a third cross-attention mechanism, the following is also included: The fourth cross-attention mechanism is used to interactively learn the prior features and the output of the third cross-attention mechanism to obtain the local instance vector; the prior features are the vectorized features of the local map.

8. The method according to any one of claims 1-7, characterized in that, The multimodal features include scalarized features, and the extraction of multimodal features from the local map includes: The local map is divided into blocks to obtain multiple block images; A multi-transformer image feature extraction network is used to extract multi-scale semantic information of each image block. The multi-scale semantic information of each image block is superimposed to obtain the scalarized features of the local map.

9. The method according to claim 8, characterized in that, The multimodal features also include vectorized features, and the extraction of multimodal features from the local map further includes: The coordinates of N points are sampled from the local map to determine the coordinates of N points; Determine the category of the N points; The coordinates and categories of the N points are concatenated to obtain the vectorized instance; Feature extraction is performed on the vectorized instance to obtain the vectorized features of the local map.

10. The method according to claim 9, characterized in that, The vectorized features include prior features, and the step of extracting features from the vectorized instances to obtain the vectorized features of the local map includes: The coordinate and category features of the vectorized instance are extracted using a position encoder and a category encoder. The coordinate features and category features are concatenated to obtain the prior features.

11. The method according to any one of claims 1-10, characterized in that, The number of local instance vectors is multiple, and a global vectorized map is obtained based on the local instance vectors, including: A global instance vector is obtained by concatenating multiple local instance vectors. The point coordinates of the global instance vector are sparsified and sampled to obtain a global vectorized map.

12. A device for multimodal vectorization mapping, characterized in that, The device includes: The map segmentation module is used to segment the global map to obtain local maps; the global map is composed of multimodal data. The feature extraction module is used to extract multimodal features from the local map; An instance vector prediction model is used to obtain local instance vectors based on the multimodal features; the local instance vectors are a data sequence of geometric semantic information of the local map. A vector stitching module is used to obtain a global vectorized map based on the local instance vectors; the global vectorized map is an image representing the geometric semantic information of the global map in vectorized form.

13. The apparatus according to claim 12, characterized in that, The instance vector prediction model includes: A multimodal feature fusion module is used to fuse the multimodal features to obtain a first fused feature; A deep supervision module is used to perform deep supervised learning on the first fused feature to obtain the second fused feature; A sparse instance decoder is used to decode the first fused feature and the second fused feature to obtain the local instance vector.

14. The apparatus according to claim 13, characterized in that, The device further includes: A pseudo-temporal feature extraction model is used to obtain multimodal features of the surrounding images of the local map; the surrounding images include the top, bottom, left, and right images of the local map; The multimodal feature fusion module is also used for The multimodal features of the local map and the multimodal features of the surrounding image are interactively learned using a first cross-attention mechanism to obtain optimized multimodal features; the optimized multimodal features are then fused to obtain the first fused feature.

15. The apparatus according to claim 13, characterized in that, The device further includes: A weighted model is used to learn hyperparameters of the multimodal features to obtain weighted weights; the weighted weights are adaptively updated according to scene changes on the local map. The multimodal feature fusion module is used to merge and encode the multimodal features according to the weighted weights to obtain the first fused feature.

16. The apparatus according to any one of claims 13-15, characterized in that, The device further includes: A dense task learning module is used to perform dense task learning based on the first fusion feature to determine the dense features of the local map, wherein the dense features include the semantic segmentation features and / or depth features of the local map; The deep supervision module is used for: The second fusion feature is obtained by interactively learning the first fused feature and the dense feature using a second cross-attention mechanism.

17. The apparatus according to any one of claims 13-16, characterized in that, The sparse instance decoder is used to perform self-learning updates of the first fused features using the self-attention mechanism; The second fused feature and the self-learning updated first fused feature are interactively learned using a third cross-attention mechanism to obtain the local instance vector.

18. The apparatus according to claim 17, characterized in that, The sparse instance decoder is further configured to use a fourth cross-attention mechanism to interactively learn the prior features and the output of the third cross-attention mechanism to obtain the local instance vector, wherein the prior features are vectorized features of the local map.

19. The apparatus according to any one of claims 12-18, characterized in that, The multimodal features include scalarized features, and the feature extraction module includes: An image feature extractor is used to divide the local map into blocks to obtain multiple block images; a multi-scale semantic information of each block image is extracted using an image feature extraction network with multiple transformers; the multi-scale semantic information of each block image is superimposed to obtain the scalarized features of the local map.

20. The apparatus according to any one of claims 12-19, characterized in that, The multimodal features also include vectorized features, and the feature extraction module is further used for: The coordinates of N points are sampled from the local map to determine the coordinates of the N points; the categories of the coordinates of the N points are determined; the vectorized instance is obtained by stitching together the coordinates and categories of the N points; and the vectorized instance is used to extract features to obtain the vectorized features of the local map.

21. The apparatus according to any one of claims 20, characterized in that, The device further includes: A position encoder is used to extract coordinate features from the vectorized instance; and A category encoder is used to extract category features from the vectorized instance; The feature extraction module is also used to concatenate the coordinate features and the position features to obtain prior features.

22. The apparatus according to any one of claims 12-21, characterized in that, The number of local instance vectors is multiple, and the vector concatenation module is used for: A global instance vector is obtained by concatenating multiple local instance vectors. The point coordinates of the global instance vector are sparsified and sampled to obtain a global vectorized map.

23. A computing device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, causing the computing device to perform the method as described in any one of claims 1-11.

24. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-11.

25. A computer-readable storage medium storing a computer program that, when executed on a processor, causes the processor to perform the method as described in any one of claims 1-11.

26. A computer program product, characterized in that, When the computer program product is run on a processor, it causes the processor to perform the method as described in any one of claims 1-11.

Images