Environment detection method and apparatus for vehicle, and device, vehicle, medium and product

Abstract

The embodiments of the present disclosure relate to an environment detection method and apparatus for a vehicle, and a device, a vehicle, a medium and a product. The method comprises: respectively performing feature extraction on a surround-view image and a panoramic image of a vehicle, so as to obtain a surround-view image feature and a panoramic image feature; on the basis of a task type corresponding to a detection task, determining first query information; on the basis of the surround-view image feature and the panoramic image feature, determining first key information and first value information; using an attention mechanism to process the first query information, the first key information and the first value information, so as to obtain a target fused feature; and on the basis of the target fused feature, detecting the environment around the vehicle, so as to determine a detection result corresponding to the detection task.

Description

Environmental testing methods, devices, equipment, vehicles, media and products for vehicles

[0001] Cross-references

[0002] This application claims priority to Chinese Patent Application No. 202510047555.X, filed on January 13, 2025, entitled “Environmental Testing Method, Apparatus, Equipment, Vehicle, Medium and Product for Vehicles”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to, but is not limited to, the field of automotive technology, such as a method, apparatus, equipment, vehicle, medium, and product for environmental testing of vehicles. Background Technology

[0004] When detecting the vehicle's surroundings, the system fully utilizes image information from various onboard perspectives to enhance the comprehensive perception and localization capabilities of targets and the environment. Supporting driving and parking functionalities, vehicles are equipped with panoramic and surround-view cameras capable of covering a 360° field of view around the vehicle, enabling detection coverage of the surroundings at close to medium distances.

[0005] Public content

[0006] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0007] This disclosure provides an environmental detection method, apparatus, equipment, vehicle, medium, and product for vehicles.

[0008] The technical solution adopted in the embodiments of this disclosure is as follows:

[0009] A vehicle environment detection method includes: extracting features from a circumferential view image and a surround view image of the vehicle to obtain circumferential view image features and surround view image features, respectively; determining first query information based on the task type corresponding to the detection task; determining first key information and first value information based on the circumferential view image features and the surround view image features; processing the first query information, first key information, and first value information using an attention mechanism to obtain target fusion features; and detecting the surrounding environment of the vehicle based on the target fusion features to determine the detection result corresponding to the detection task.

[0010] Based on the aforementioned technical means, features are extracted from the vehicle's perimeter and circumferential images to obtain perimeter image features and circumferential image features, respectively. Then, according to the task type corresponding to the detection task, the first query information is determined, and based on the perimeter and circumferential image features, the first key information and first value information corresponding to the detection task are determined. Using an attention mechanism, the first query information, first key information, and first value information are processed to obtain target fusion features. Then, based on the target fusion features, the vehicle's surrounding environment is detected to determine the detection result corresponding to the detection task. In this way, on the one hand, the attention mechanism is used to integrate the input and output of perimeter and ring image features, reducing resource contention and latency caused by independent operation, effectively improving information processing efficiency. Furthermore, by constructing key and value information in the attention mechanism using perimeter and ring image features, the perimeter and ring image features can be better integrated and utilized, thereby improving the perception capability in the environmental detection process and increasing the accuracy of environmental detection. On the other hand, the attention mechanism is used to sparsely represent task-related information, perimeter and ring image features, thereby reducing the computational load and memory consumption in the feature fusion process.

[0011] Optionally, the panoramic image features include two-dimensional panoramic features and three-dimensional panoramic position features, and the 3D surround view image features include two-dimensional surround view features and three-dimensional surround view position features; feature extraction is performed on the panoramic image and the 3D surround view image of the vehicle respectively to obtain the panoramic image features and the 3D surround view image features, including: performing two-dimensional encoding on the panoramic image and the 3D surround view image to determine the two-dimensional panoramic features and the two-dimensional surround view features; and performing three-dimensional encoding on the panoramic image and the 3D surround view image to determine the three-dimensional panoramic position features and the three-dimensional surround view position features.

[0012] Based on the aforementioned technical methods, the panoramic image features include two-dimensional panoramic features and three-dimensional panoramic position features, while the all-around image features include two-dimensional all-around features and three-dimensional all-around position features. Two-dimensional encoding is performed on the panoramic and all-around images to determine their respective two-dimensional panoramic and all-around features; similarly, three-dimensional encoding is performed on the panoramic and all-around images to determine their respective three-dimensional panoramic and all-around position features. This allows the panoramic and all-around image features to include complete two-dimensional image information and three-dimensional spatial position information, improving the accuracy of environmental detection.

[0013] Optionally, based on the task type corresponding to the detection task, the first query information is determined, including: based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment, the second query information, the second key information, and the second value information corresponding to the detection task are determined; and, using an attention mechanism, the second query information, the second key information, and the second value information are processed to obtain the first query information.

[0014] Based on the above technical means, by determining the second query information, second key information and second value information corresponding to the detection task according to the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical time, the second query information, second key information and second value information are determined; by using the attention mechanism, the second query information, second key information and second value information are processed to obtain the first query information, which can fuse the task-related information at the current time and the task-related information at each historical time.

[0015] Optionally, based on the task type corresponding to the detection task and the detection results corresponding to the detection tasks at each historical moment, the second query information, second key information, and second value information corresponding to the detection task are determined, including: reconstructing the detection results corresponding to the detection tasks at each historical moment based on the pose transformation relationship between each historical moment and the current moment, and determining the reconstruction features corresponding to the detection tasks at each historical moment; determining the initial reconstruction features corresponding to the detection task at the current moment based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection tasks at the first historical moment; wherein, the first historical moment is the moment before the current moment; determining the initial reconstruction features corresponding to the detection task at the current moment as the second query information; and determining the second key information and second value information based on the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment; wherein, the second historical moment is a historical moment other than the first historical moment.

[0016] Based on the aforementioned technical means, and based on the pose transformation relationship between the vehicle at each historical moment and the current moment, the detection results corresponding to the detection tasks at each historical moment are reconstructed to determine the reconstruction features corresponding to the detection tasks at each historical moment. Furthermore, based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment, the initial reconstruction features corresponding to the detection task at the current moment are determined; the first historical moment is the previous moment of the current moment. Then, the initial reconstruction features corresponding to the detection task at the current moment are determined as the second query information. Subsequently, based on the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment, second key information and second value information are determined; the second historical moments are historical moments other than the first historical moment. Thus, the second query information includes the initial reconstruction features corresponding to the detection task at the current moment, and the second key information and second value information include the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment. This allows for the fusion of task-related information from the current moment and task-related information from each historical moment into the first query information, thereby establishing a correlation between task-related information from the current moment and task-related information from each historical moment in the target fusion features, thereby improving the accuracy of environmental detection.

[0017] Optionally, based on the pose transformation relationship between the vehicle at each historical moment and the current moment, the detection results corresponding to the detection task at each historical moment are reconstructed to determine the reconstruction features corresponding to the detection task at each historical moment. This includes: aligning the detection results corresponding to the detection task at each historical moment to the current moment based on the pose transformation relationship to determine the updated detection results corresponding to the detection task at each historical moment; and reconstructing the updated detection results corresponding to the detection task at each historical moment based on the pose transformation relationship, the time difference between each historical moment and the current moment, and the motion speed corresponding to the detection task to determine the reconstruction features corresponding to the detection task at each historical moment.

[0018] Based on the aforementioned technical means, by aligning the detection results corresponding to the detection tasks at each historical moment to the current moment according to the pose transformation relationship, the updated detection results corresponding to the detection tasks at each historical moment are determined. Then, based on the pose transformation relationship, the time difference between each historical moment and the current moment, and the motion speed corresponding to the detection task, the updated detection results corresponding to the detection tasks at each historical moment are reconstructed, determining the reconstructed features corresponding to the detection tasks at each historical moment. In this way, aligning and reconstructing the detection results corresponding to the detection tasks at historical moments to the current moment ensures that the task-related information at the current moment and the task-related information at historical moments maintain semantic continuity and consistency.

[0019] Optionally, based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment, the initial reconstruction features corresponding to the detection task at the current moment are determined, including: based on the reconstruction features corresponding to the detection task at the first historical moment, determining the first initial reconstruction features corresponding to the detection task at the current moment; based on the task type corresponding to the detection task, randomly generating the second initial reconstruction features corresponding to the detection task at the current moment; and concatenating the first initial reconstruction features and the second initial reconstruction features to determine the initial reconstruction features corresponding to the current moment.

[0020] Based on the aforementioned technical means, a first initial reconstruction feature corresponding to the detection task at the current moment is determined according to the reconstruction feature corresponding to the detection task at the first historical moment. Then, a second initial reconstruction feature corresponding to the detection task at the current moment is randomly generated according to the task type. Finally, the first and second initial reconstruction features are concatenated to determine the initial reconstruction feature corresponding to the current moment. In this way, on the one hand, initializing using the reconstruction feature corresponding to the detection task at the first historical moment can maintain a certain continuity between the task-related information at the current moment and the task-related information at historical moments, helping to reduce instability caused by random initialization. On the other hand, random initialization allows for the learning of more diverse information.

[0021] Optionally, the step of randomly generating the second initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task includes: randomly generating the initial feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task; and reconstructing the initial feature corresponding to the task type to determine the second initial reconstruction feature corresponding to the detection task at the current moment.

[0022] Based on the aforementioned technical methods, initial features corresponding to the detection task at the current moment are randomly generated according to the task type. These initial features are then reconstructed to determine the second initial reconstructed features corresponding to the detection task at the current moment. This process maintains consistency between the reconstructed initial features and those from previous historical moments.

[0023] Optionally, based on the task type corresponding to the detection task, initial features corresponding to the detection task at the current moment are randomly generated, including at least one of the following: when the detection task includes a moving target detection task, the coordinates in three-dimensional space are uniformly sampled to determine the initial features corresponding to the moving target detection task; when the detection task includes a road segmentation detection task, the initial features corresponding to the road segmentation detection task are determined based on the coordinates of the center positions of each sub-region in the set vehicle passable area; and when the detection task includes a lane line detection task, the coordinates in two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane line detection task.

[0024] Based on the aforementioned technical methods, when the detection task includes moving target detection, the coordinates in three-dimensional space are uniformly sampled to determine the initial features corresponding to the moving target detection task. When the detection task includes road segmentation detection, the initial features corresponding to the road segmentation detection task are determined based on the coordinates of the center positions of each sub-region within the defined vehicle-passable area. When the detection task includes lane line detection, the coordinates in two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane line detection task. In this way, different initialization methods are used for different detection tasks, improving the accuracy of initialization.

[0025] Optionally, determining the first key information and first value information corresponding to the detection task based on the perimeter image features and the surrounding image features includes: concatenating the two-dimensional perimeter features and the two-dimensional surrounding features to determine the first key information corresponding to the detection task; and concatenating the two-dimensional perimeter features, the three-dimensional perimeter position features, the two-dimensional surrounding features, and the three-dimensional surrounding position features to determine the first value information corresponding to the detection task.

[0026] Based on the aforementioned technical means, the two-dimensional panoramic features and the two-dimensional surround-view features are stitched together to determine the first key information corresponding to the detection task; the two-dimensional panoramic features, the three-dimensional panoramic position features, the two-dimensional surround-view features, and the three-dimensional surround-view position features are stitched together to determine the first value information corresponding to the detection task. This ensures that the first value information and the first key information include complete panoramic and surround-view image features, improving the accuracy of detecting the vehicle's surrounding environment.

[0027] Optionally, an attention mechanism is used to process the first query information, the first key information, and the first value information to obtain target fusion features, including: using the first query information as the query matrix of the attention mechanism and the first key information as the key matrix of the attention mechanism to generate an attention matrix; and determining the target fusion features based on the attention matrix and the first value information.

[0028] Based on the aforementioned technical means, the first query information is used as the query matrix of the attention mechanism, and the first key information is used as the key matrix of the attention mechanism to generate an attention matrix. Based on the attention matrix and the first key information, the target fusion features are determined. In this way, on the one hand, the attention mechanism is used to sparsely represent task-related information, perimeter image features, and ring-view image features, thereby reducing computational load and memory consumption; on the other hand, the attention mechanism is used to fuse perimeter image features and ring-view image features, and to establish the correlation between each historical moment and the current moment, thereby improving the accuracy of detecting the vehicle's surrounding environment.

[0029] Optionally, there are multiple detection tasks; the first query information is determined based on the task type corresponding to the detection task, including: for each detection task, determining the query information corresponding to the detection task based on the task type corresponding to the detection task; and concatenating the query information corresponding to each detection task to determine the first query information.

[0030] Based on the aforementioned technical methods, for each detection task, the corresponding query information is determined according to the task type. The query information from each detection task is then concatenated to determine the first query information. In this way, multi-task fusion achieves comprehensive perception of the road environment, improving the accuracy and robustness of vehicle surrounding environment detection.

[0031] Optionally, the method further includes: extracting features from the radar point cloud data of the vehicle to obtain point cloud features; and determining first key information and first value information based on the circumferential image features and the ring image features, including: determining first key information and first value information based on the circumferential image features, the ring image features and the point cloud features.

[0032] Based on the above technical means, feature extraction is performed on the radar point cloud data of the vehicle to obtain point cloud features. Then, based on the perimeter image features, the ring image features, and the point cloud features, the first key information and the first value information are determined so that the first key information and the first value information include the purely visual perimeter image features and the ring image features, as well as the point cloud features, thereby improving the accuracy of detecting the vehicle's surrounding environment.

[0033] An environmental detection device for a vehicle, the environmental detection device comprising:

[0034] The feature extraction unit is configured to extract features from the circumferential view image and the surround view image of the vehicle, respectively, to obtain the circumferential view image features and the surround view image features;

[0035] The first determining unit is configured to determine the first query information based on the task type corresponding to the detection task;

[0036] The second determining unit is configured to determine the first key information and the first value information corresponding to the detection task based on the features of the perimeter image and the features of the ring image;

[0037] The processing unit is configured to use an attention mechanism to process the first query information, the first key information, and the first value information to obtain the target fusion feature; and

[0038] The third determination unit is configured to detect the vehicle's surrounding environment based on target fusion features and determine the detection results corresponding to the detection task.

[0039] This disclosure provides a computer device including a memory and a processor. The memory stores computer instructions that can be executed on the processor. When the processor executes the instructions, it implements some or all of the steps in the above-described method.

[0040] This disclosure provides a vehicle that includes the computer equipment described above.

[0041] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement some or all of the steps in the above-described method.

[0042] This disclosure provides a computer program product, including computer instructions, which, when executed by a processor, implement some or all of the steps in the above-described method.

[0043] After reading and understanding the accompanying diagrams and detailed descriptions, the other aspects can be understood. Attached Figure Description

[0044] Figure 1 is a schematic diagram of the implementation process of a vehicle environmental detection method proposed in an embodiment of this disclosure;

[0045] Figure 2 is a schematic diagram of a vehicle environment detection structure using a sparse HAS-Propagation Transformer architecture proposed in an embodiment of this disclosure.

[0046] Figure 3 is a schematic diagram of a network structure for vehicle environment detection using a sparse HAS-Propagation Transformer architecture proposed in an embodiment of this disclosure.

[0047] Figure 4 is a schematic diagram of the implementation process of a BEV surrounding view dynamic and static multi-task perception method based on a sparse HAS-Propagation Transformer architecture proposed in an embodiment of this disclosure.

[0048] Figure 5 is a schematic diagram of a camera coordinate system and a vehicle coordinate system proposed in an embodiment of this disclosure;

[0049] Figure 6 is a schematic diagram of a historical cache information structure proposed in an embodiment of this disclosure;

[0050] Figure 7 is a schematic diagram of historical cache embedding reconstruction for a 3D object detection task according to an embodiment of this disclosure;

[0051] Figure 8 is a schematic diagram of historical cache embedding reconstruction for a road segmentation detection task proposed in an embodiment of this disclosure.

[0052] Figure 9 is a schematic diagram of historical cache embedding reconstruction for a lane line detection task according to an embodiment of this disclosure.

[0053] Figure 10 is a schematic diagram of the physical objects for each detection task proposed in an embodiment of this disclosure;

[0054] Figure 11 is a schematic diagram of a HAS-Propagation Transformer structure proposed in an embodiment of this disclosure;

[0055] Figure 12 is a schematic diagram of the composition structure of a vehicle environmental detection device according to an embodiment of this disclosure;

[0056] Figure 13 is a schematic diagram of the hardware entity of a computer device according to an embodiment of this disclosure. Detailed Implementation

[0057] The embodiments of this disclosure will be described below with reference to the accompanying drawings and examples. Those skilled in the art can understand other advantages and effects of the embodiments of this disclosure from the content disclosed in this specification. The embodiments of this disclosure can also be implemented or applied through other different implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the embodiments of this disclosure. The embodiments are only for illustrating the implementation schemes of this disclosure and are not intended to limit the protection scope of the embodiments of this disclosure.

[0058] The illustrations provided in the following embodiments are only schematic representations of the basic concept of the embodiments of this disclosure. Therefore, the drawings only show the components related to the embodiments of this disclosure and are not drawn according to the actual number, shape and size of the components. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0059] In implementing the embodiments of this disclosure, it was found that due to the significant differences between panoramic and 360-degree view images in terms of field of view and distortion mode, panoramic and 360-degree view images are generally processed independently for driving and parking tasks, resulting in the consumption of limited vehicle resources and high latency caused by preemption, as well as the inability to effectively compensate for the inherent blind spot characteristics of panoramic view.

[0060] This disclosure provides a vehicle environmental detection method, as shown in Figure 1. The vehicle environmental detection method includes the following steps S101 to S105, wherein:

[0061] Step S101: Extract features from the circumferential view image and the 360-degree view image of the vehicle respectively to obtain the circumferential view image features and the 360-degree view image features;

[0062] A panoramic image is an image that can show the environment surrounding a certain area or object. This image can be a visual presentation radiating outwards from a central point. Panoramic images can capture high-definition details of the surrounding environment, but the coverage area is limited.

[0063] Among them, panoramic images can capture and display a 360-degree panoramic image of a scene or object, with a large coverage area, but with greater distortion.

[0064] In some embodiments, a panoramic image is acquired using a panoramic camera, such as a pinhole camera; and a surround image is acquired using a 360-degree camera, such as a fisheye camera.

[0065] In some embodiments, the panoramic image features include two-dimensional panoramic features and three-dimensional panoramic position features.

[0066] In some embodiments, the surround view image features include two-dimensional surround view features and three-dimensional surround view position features.

[0067] Step S102: Determine the first query information based on the task type corresponding to the detection task;

[0068] In some embodiments, the detection task may include a moving target detection task, a road segmentation detection task, and a lane line detection task.

[0069] In some embodiments, different detection tasks are different task types.

[0070] In some embodiments, the first query information is information related to the detection task.

[0071] In some embodiments, the query matrix, key matrix, and value matrix can be determined according to the task type corresponding to the detection task, and the first query information can be determined using a hybrid attention mechanism.

[0072] Step S103: Based on the perimeter image features and the ring view image features, determine the first key information and the first value information;

[0073] In some embodiments, the first key information includes panoramic two-dimensional features and all-around two-dimensional features.

[0074] In some embodiments, the first value information includes panoramic two-dimensional features, panoramic three-dimensional features, surround two-dimensional features, and surround three-dimensional features.

[0075] Step S104: Using an attention mechanism, process the first query information, the first key information, and the first value information to obtain target fusion features;

[0076] In some embodiments, the first query information is used as the query matrix, the first key information is used as the key matrix, and the first value information is used as the value matrix. The target fusion features are determined by using a panoramic attention mechanism.

[0077] Step S105: Based on the target fusion features, detect the surrounding environment of the vehicle and determine the detection result corresponding to the detection task.

[0078] In some embodiments, the target fusion features corresponding to the detection task are connected to the corresponding decoding head network to obtain multiple decoding results corresponding to the detection task. The results are sorted according to the decoding confidence level of the decoding results, and the target decoding results of the target number are selected from the multiple decoding results. Based on the target decoding results, the detection results corresponding to the detection task are determined.

[0079] In some embodiments, the sorted decoding results are stored in the historical cache queue in a first-in-first-out (FIFO) manner for processing in the next step.

[0080] In some embodiments, the decoding of the moving target detection task can employ a simple multilayer perceptron network with an input dimension of C and an output dimension that includes information such as target confidence, target category, center position, bounding box dimensions, target orientation, and target velocity, which can be set according to actual needs.

[0081] In some embodiments, the decoding of the road segmentation detection task can segment pixel-level location regions on the BEV plane, since each road query corresponds to the Bird's Eye View (BEV) plane S. H ×S W One of a number of rectangular facets, each facet corresponding to r h ×r w Pixel size, therefore, an input dimension of C and an output dimension of r can be used. h ×r w A simple multilayer perceptron network with added 1-dimensional confidence; or a simple CNN network can be used to evaluate r. h ×r w Perform pixel segmentation.

[0082] In some embodiments, the lane line detection task employs a multilayer perceptron network with an input dimension of C, but its output dimension depends on the vectorized representation of the lane line. Typical representations include MapTR, PETRv2, and even the 3D lane line PersFormer method (such as using 3D lane lines). The appropriate method can be chosen based on actual needs and complexity constraints, and this disclosure does not impose any limitations.

[0083] In this embodiment, features are extracted from the vehicle's perimeter and surrounding images to obtain perimeter image features and surrounding image features, respectively. Then, based on the task type corresponding to the detection task, first query information is determined, and based on the perimeter and surrounding image features, first key information and first value information corresponding to the detection task are determined. An attention mechanism is used to process the first query information, first key information, and first value information to obtain target fusion features. Then, based on the target fusion features, the vehicle's surrounding environment is detected, and the detection result corresponding to the detection task is determined. Thus, on the one hand, the attention mechanism integrates the perimeter and surrounding image features into a unified input and output, reducing resource contention and latency caused by independent operation, effectively improving information processing efficiency. Furthermore, by constructing the key and value information in the attention mechanism using the perimeter and surrounding image features, the perimeter and surrounding image features can be better integrated and utilized, thereby improving the perception capability during environmental detection and increasing the accuracy of environmental detection. On the other hand, the attention mechanism uses sparse representation of task-related information, perimeter image features, and surrounding image features, thereby reducing the computational load and memory consumption during feature fusion.

[0084] In some embodiments, the first key information and the first value information can be determined based on the perimeter image features and the surrounding image features. In this way, by fusing the purely visual perimeter image features and the surrounding image features, a target fusion feature for detecting the vehicle's surrounding environment can be obtained, reducing the high cost of using other non-visual acquisition devices (such as LiDAR) for detection, thereby lowering the cost of detecting the vehicle's surrounding environment.

[0085] In some embodiments, the first key information and the first value information can be determined based on the perimeter image features, the surrounding image features, and non-visual modal features obtained by feature extraction from non-visual modal perception data of the vehicle (such as radar point cloud data). In this way, by fusing the perimeter and surrounding image features under the visual modality, as well as the non-visual modal features, the perception capability of the environment can be improved, thereby enhancing the accuracy of detecting the vehicle's surrounding environment.

[0086] In some embodiments, the circumferential image features include two-dimensional circumferential features and three-dimensional circumferential position features, and the surround image features include two-dimensional surround image features and three-dimensional surround position features; the feature extraction of the circumferential image and surround image of the vehicle in step S101 to obtain the circumferential image features and surround image features respectively may include the following steps S1011 and S1012, wherein:

[0087] Step S1011: Perform two-dimensional encoding on the perimeter view image and the toroidal view image to determine the two-dimensional perimeter view features and the two-dimensional toroidal view features;

[0088] In some embodiments, two independent 2D convolutional coding backbone networks are used to encode the perimeter view image and the toroidal view image respectively to obtain two-dimensional perimeter view features and two-dimensional toroidal view features.

[0089] In some embodiments, the backbone network may typically adopt, but is not limited to, a series of commonly used 2D convolutional network structures with a certain depth, such as ResNet, EfficientNet, SwinTransformer, and VoVNetV2.

[0090] In some embodiments, all perimeter images and all ring images each share a set of network weights.

[0091] Step S1012: Perform three-dimensional encoding on the circumferential view image and the surround view image to determine the three-dimensional circumferential position features and the three-dimensional surround view position features.

[0092] In some embodiments, based on the distortion model of the panoramic camera and the distortion parameters corresponding to the distortion model, combined with the intrinsic parameters of the panoramic camera, the distortion-free true vector direction corresponding to each pixel in the panoramic image in the panoramic camera coordinate system is calculated. Then, reasonable equally spaced depth segments are set in the BEV space, and the coordinate values ​​of each pixel in the image in different depth segments in the panoramic camera coordinate system are constructed by combining the vector directions of each pixel. Then, using the extrinsic rotation and displacement information of the panoramic camera relative to the vehicle coordinate system, the coordinates of each pixel in different depth segments are transformed to the vehicle coordinate system. After that, Reference Positional Encoding (RefPE) is used to encode the coordinates of each pixel in the panoramic image in the vehicle coordinate system corresponding to different depth segments to obtain the three-dimensional panoramic position features.

[0093] In some embodiments, based on the distortion model of the surround-view camera and the distortion parameters corresponding to the distortion model, combined with the intrinsic parameters of the surround-view camera, the distortion-free true vector direction corresponding to each pixel in the surround-view image in the surround-view camera coordinate system is calculated. Then, reasonable equally spaced depth segments are set in the BEV space, and the coordinate values ​​of each pixel in the image in different depth segments in the surround-view camera coordinate system are constructed by combining the vector directions of each pixel. Then, using the extrinsic rotation and displacement information of the surround-view camera relative to the vehicle coordinate system, the coordinates of each pixel in different depth segments are transformed to the vehicle coordinate system. After that, Reference Positional Encoding (RefPE) is used to encode the coordinates of each pixel in the surround-view image in the vehicle coordinate system corresponding to different depth segments to obtain the three-dimensional surround-view position features.

[0094] In this embodiment, the panoramic image features include two-dimensional panoramic features and three-dimensional panoramic position features, and the all-around image features include two-dimensional all-around features and three-dimensional all-around position features. Two-dimensional encoding is performed on the panoramic and all-around images to determine the two-dimensional panoramic and all-around features; three-dimensional encoding is then performed on the panoramic and all-around images to determine the three-dimensional panoramic and all-around position features. This allows the panoramic and all-around image features to include complete two-dimensional image information and three-dimensional spatial position information, improving the accuracy of environmental detection.

[0095] In some embodiments, determining the first query information based on the task type corresponding to the detection task in step S102 may include the following steps S1021 and S1022, wherein:

[0096] Step S1021: Based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical time, determine the second query information, second key information and second value information corresponding to the detection task;

[0097] In some embodiments, the initial features at the current time are initialized according to the task type corresponding to the detection task, and the second query information corresponding to the detection task is determined according to the detection result of the detection task at the previous time and the initial features at the current time.

[0098] In some embodiments, the second key information and the second value information are determined based on the initial features at the current time and the detection results corresponding to the detection tasks at each historical time; the second key information and the second value information are consistent.

[0099] Step S1022: Using the attention mechanism, process the second query information, the second key information, and the second value information to obtain the first query information.

[0100] In some embodiments, the second query information is used as a query matrix, the second key information is used as a key matrix, and the second value information is used as a value matrix. These are input into the hybrid attention mechanism to output the first query information.

[0101] In this embodiment of the disclosure, the second query information, the second key information, and the second value information corresponding to the detection task are determined according to the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical time. The second query information, the second key information, and the second value information are processed using an attention mechanism to obtain the first query information, which can fuse the task-related information at the current time and the task-related information at each historical time.

[0102] In some embodiments, the step S1021, which involves determining the second query information, second key information, and second value information corresponding to the detection task based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical time, may include the following steps S10211 to S10214, wherein:

[0103] Step S10211: Based on the pose transformation relationship of the vehicle at each historical moment and the current moment, reconstruct the detection results corresponding to the detection task at each historical moment, and determine the reconstruction features corresponding to the detection task at each historical moment;

[0104] In some embodiments, the decoding results corresponding to each historical moment and the detection task are read from the historical cache queue, and the decoding results corresponding to each historical moment and the detection task are aligned to the current moment to obtain the updated decoding results corresponding to each historical moment and the detection task at the current moment.

[0105] In some embodiments, the updated decoding results corresponding to the detection tasks at each historical moment are reconstructed to generate a high-dimensional reconstruction embedding vector spatially aligned with the current moment for each historical moment.

[0106] Step S10212: Based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment, determine the initial reconstruction features corresponding to the detection task at the current moment; the first historical moment is the previous moment of the current moment;

[0107] In some embodiments, the reconstructed features corresponding to the detection task at the first historical moment are propagated to initialize the initial reconstructed features at the current moment, thereby obtaining the first initial reconstructed features.

[0108] In some embodiments, multiple second initial reconstruction features are randomly generated based on the task type corresponding to the detection task.

[0109] In some embodiments, the first initial reconstruction feature and the second initial reconstruction feature are concatenated to obtain the initial reconstruction feature corresponding to the detection task at the current time.

[0110] In some embodiments, the initial reconstruction features corresponding to the moving target detection task include the initial position and context features of the moving target in the BEV space; the initial reconstruction features corresponding to the road segmentation detection task include the BEV ground plane S H ×S W The context features corresponding to each rectangular patch of equal size are fixed at the center point of each patch; the initial reconstruction features corresponding to the lane line detection task include the initial geometric center position and context features of each lane line or road edge.

[0111] Step S10213: Determine the initial reconstruction feature corresponding to the detection task at the current time as the second query information;

[0112] Step S10214: Based on the reconstruction features corresponding to the detection task at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment, determine the second key information and the second value information; the second historical moment is a historical moment other than the first historical moment.

[0113] In some embodiments, the reconstructed features corresponding to each second historical moment detection task and the initial reconstructed features corresponding to the current moment detection task are concatenated to obtain second key information and second value information; the second key information and the second value information are consistent.

[0114] In this embodiment, based on the pose transformation relationship between the vehicle at each historical moment and the current moment, the detection results corresponding to the detection tasks at each historical moment are reconstructed to determine the reconstruction features corresponding to the detection tasks at each historical moment. Furthermore, based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment, the initial reconstruction features corresponding to the detection task at the current moment are determined; the first historical moment is the previous moment of the current moment. Then, the initial reconstruction features corresponding to the detection task at the current moment are determined as the second query information. Next, based on the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment, second key information and second value information are determined; the second historical moments are historical moments other than the first historical moment. Thus, the second query information includes the initial reconstruction features corresponding to the detection task at the current moment. This allows the fusion of task-related information from the current moment and task-related information from each historical moment into the first query information, thereby establishing a correlation between task-related information from the current moment and task-related information from each historical moment in the target fusion features, thereby improving the accuracy of environmental detection.

[0115] In some embodiments, the step S10211, which involves reconstructing the detection results corresponding to the detection task at each historical time based on the pose transformation relationship between the vehicle at each historical time and the current time, and determining the reconstruction features corresponding to the detection task at each historical time, may include the following steps S102111 and S102112, wherein:

[0116] Step S102111: Based on the pose transformation relationship, align the detection results corresponding to the detection tasks at each of the historical moments to the current moment, and determine the updated detection results corresponding to the detection tasks at each of the historical moments;

[0117] In some embodiments, based on the pose transformation relationship, the detection results corresponding to the moving target detection task, road segmentation detection task, and lane line detection task stored at each historical moment are transformed and aligned to the current vehicle coordinate system.

[0118] In some embodiments, since the center position corresponding to the road segmentation detection task and the lane line detection task has a dimension of 2, it is necessary to supplement the height value of the ground in the vehicle coordinate system in the height Z-axis direction to expand it to 3-dimensional, and then calculate the coordinates of the center position corresponding to the road segmentation detection task and the lane line detection task at the current time according to the conversion formula.

[0119] Step S102112: Based on the pose transformation relationship, the time difference between each historical moment and the current moment, and the motion speed corresponding to the detection task, reconstruct the updated detection results corresponding to the detection task at each historical moment, and determine the reconstruction features corresponding to the detection task at each historical moment.

[0120] In some embodiments, the pose transformation relationship, time difference, and target motion speed are used to calculate the MLN correspondence coefficients according to the motion perception regularization (MLN) method, and the detection results cached at each historical time are aligned to the current time embedding for reconstruction using the MLN correspondence coefficients.

[0121] In this embodiment, by aligning the detection results corresponding to the detection tasks at each historical moment to the current moment according to the pose transformation relationship, the updated detection results corresponding to the detection tasks at each historical moment are determined. Based on the pose transformation relationship, the time difference between each historical moment and the current moment, and the motion speed corresponding to the detection task, the updated detection results corresponding to the detection tasks at each historical moment are reconstructed to determine the reconstructed features corresponding to the detection tasks at each historical moment. In this way, aligning the detection results corresponding to the detection tasks at historical moments to the current moment and reconstructing them ensures that the task-related information at the current moment and the task-related information at historical moments maintain semantic continuity and consistency.

[0122] In some embodiments, the step S10212, which involves determining the initial reconstruction features corresponding to the detection task at the current time based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical time, may include the following steps S102121 to S102123, wherein:

[0123] Step S102121: Based on the reconstruction features corresponding to the detection task at the first historical moment, determine the first initial reconstruction features corresponding to the detection task at the current moment;

[0124] In some embodiments, the reconstructed features corresponding to the detection task at the first historical moment are propagated to obtain the first initial reconstructed features corresponding to the detection task at the current moment.

[0125] Step S102122: Based on the task type corresponding to the detection task, randomly generate the second initial reconstruction feature corresponding to the detection task at the current time;

[0126] Step S102123: Concatenate the first initial reconstruction feature and the second initial reconstruction feature to determine the initial reconstruction feature corresponding to the current time.

[0127] In some embodiments, the moving target detection task and the lane line detection task directly concatenate the first initial reconstruction feature and the second initial reconstruction feature to obtain the initial reconstruction feature.

[0128] In some embodiments, after the road segmentation detection task stitches together the first initial reconstruction feature and the second initial reconstruction feature, it also needs to confirm the S on the 2D ground plane of the BEV space in the current vehicle coordinate system corresponding to the Embedding reconstructed at the first historical moment. H ×S W The position within a rectangular block.

[0129] In this embodiment, a first initial reconstruction feature corresponding to the detection task at the current moment is determined based on the reconstruction feature corresponding to the detection task at the first historical moment. Then, a second initial reconstruction feature corresponding to the detection task at the current moment is randomly generated based on the task type corresponding to the detection task. Finally, the first and second initial reconstruction features are concatenated to determine the initial reconstruction feature corresponding to the current moment. This approach achieves two advantages: firstly, initialization using the reconstruction feature corresponding to the detection task at the first historical moment maintains a certain continuity between task-related information at the current moment and task-related information at historical moments, helping to reduce instability caused by random initialization; secondly, random initialization allows for the learning of more diverse information.

[0130] In some embodiments, the step S102122, which involves randomly generating the second initial reconstruction feature corresponding to the detection task at the current time based on the task type corresponding to the detection task, may include the following steps S1021221 and S1021222, wherein:

[0131] Step S1021221: Based on the task type corresponding to the detection task, randomly generate the initial features corresponding to the detection task at the current time.

[0132] In some embodiments, initial features corresponding to the detection task are generated based on different task types.

[0133] In some embodiments, the initial value of the context embedding vector for each detection task is set to a 0 vector.

[0134] Step S1021222: Reconstruct the initial features corresponding to the detection task to determine the second initial reconstructed features corresponding to the detection task at the current time.

[0135] In this embodiment, based on the task type corresponding to the detection task, initial features corresponding to the current detection task are randomly generated; the initial features corresponding to the detection task are reconstructed to determine the second initial reconstructed features corresponding to the current detection task. In this way, the reconstruction of the initial features corresponding to the detection task maintains consistency with the features at each historical time.

[0136] In some embodiments, the step S1021221, which involves randomly generating the initial feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task, may include at least one of the following steps S10212211 to S1021223:

[0137] Step S1611: If the detection task includes a moving target detection task, uniformly sample the coordinates in the three-dimensional space to determine the initial features corresponding to the moving target detection task;

[0138] In some embodiments, the initialization of the center position corresponding to the moving target detection task adopts the uniformly distributed coordinates of the position in the 3D space under the BEV space in the vehicle coordinate system.

[0139] Step S1612: If the detection task includes a road segmentation detection task, determine the initial features corresponding to the road segmentation detection task based on the coordinates of the center positions of each sub-region in the set vehicle passable area.

[0140] In some embodiments, the center position initialization corresponding to the road segmentation detection task uses S H ×S W The coordinates of the center position of the remaining facet after the propagation at the first historical moment in each rectangular facet (i.e., sub-region).

[0141] Step S1613: If the detection task includes a lane line detection task, uniformly sample the coordinates in the two-dimensional space to determine the initial features corresponding to the lane line detection task.

[0142] In some embodiments, the initial position of the lane detection task center is initialized using uniformly distributed coordinates on a 2D ground plane in the BEV space under the vehicle coordinate system.

[0143] In this embodiment, when the detection task includes a moving target detection task, the coordinates in three-dimensional space are uniformly sampled to determine the initial features corresponding to the moving target detection task; when the detection task includes a road segmentation detection task, the initial features corresponding to the road segmentation detection task are determined based on the coordinates of the center positions of each sub-region within a defined vehicle-passable area; when the detection task includes a lane line detection task, the coordinates in two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane line detection task. Thus, different initialization methods are used for different detection tasks, improving the accuracy of initialization.

[0144] In some embodiments, determining the first key information and the first value information based on the perimeter image features and the ring image features in step S103 may include the following steps S1031 and S1032, wherein:

[0145] Step S1031: Concatenate the two-dimensional panoramic features and the two-dimensional all-around features to determine the first key information corresponding to the detection task;

[0146] Step S1032: The two-dimensional panoramic feature, the three-dimensional panoramic position feature, the two-dimensional surround view feature, and the three-dimensional surround view position feature are spliced ​​together to determine the first value information corresponding to the detection task.

[0147] In this embodiment, the two-dimensional panoramic features and the two-dimensional surround view features are concatenated to determine the first key information corresponding to the detection task; the two-dimensional panoramic features, the three-dimensional panoramic position features, the two-dimensional surround view features, and the three-dimensional surround view position features are concatenated to determine the first value information corresponding to the detection task. This ensures that the first value information and the first key information include complete panoramic and surround view image features, improving the accuracy of detecting the vehicle's surrounding environment.

[0148] In some embodiments, step S104, which involves processing the first query information, the first key information, and the first value information using an attention mechanism to obtain target fusion features, may include the following steps S1041 and S1042, wherein:

[0149] Step S1041: Use the first query information as the query matrix of the attention mechanism and the first key information as the key matrix of the attention mechanism to generate an attention matrix;

[0150] Step S1042: Determine the target fusion feature based on the attention matrix and the first value information.

[0151] In this embodiment, the first query information is used as the query matrix of the attention mechanism, and the first key information is used as the key matrix of the attention mechanism to generate an attention matrix. Based on the attention matrix and the first value information, the target fusion features are determined. In this way, on the one hand, the attention mechanism is used to sparsely represent task-related information, perimeter image features, and ring image features, thereby reducing the amount of computation and memory usage; on the other hand, the attention mechanism is used to fuse perimeter image features and ring image features, and to establish the correlation between each historical moment and the current moment, so as to improve the accuracy of vehicle surrounding environment detection.

[0152] In some embodiments, the number of detection tasks is multiple; the step S102, which determines the first query information based on the task type corresponding to the detection task, may include the following steps S1023 and S1024, wherein:

[0153] Step S1023: For each detection task, determine the query information corresponding to the detection task based on the task type corresponding to the detection task;

[0154] In some embodiments, the detection task may include at least one of a moving target detection task, a road segmentation detection task, a lane line detection task, etc.

[0155] In some embodiments, each detection task corresponds to different query information.

[0156] Step S1024: Concatenate the query information corresponding to each of the detection tasks to determine the first query information.

[0157] In this embodiment, for each detection task, query information corresponding to the detection task is determined based on the task type. The query information corresponding to each detection task is then concatenated to determine the first query information. In this way, multi-task fusion achieves comprehensive perception of the road environment, improving the accuracy and robustness of vehicle surrounding environment detection.

[0158] In some embodiments, the above method may further include the following step S106, wherein:

[0159] Step S106: Extract features from the radar point cloud data of the vehicle to obtain point cloud features;

[0160] In some embodiments, the radar scans the environment around the vehicle and collects reflected signals to obtain radar point cloud data. The radar point cloud data provides high-precision three-dimensional perception information of the environment around the vehicle, which can accurately reflect the 3D spatial depth position of objects and the environment. It has unparalleled advantages over vision in ranging and precise positioning of spatial objects. For example, it has a superior detection capability to avoid obstacles that suddenly intrude or are not on the whitelist.

[0161] The step S103 above, which involves determining the first key information and the first value information based on the perimeter image features and the ring view image features, may include the following step S1033, wherein:

[0162] Step S1033: Based on the perimeter image features, the ring view image features, and the point cloud features, determine the first key information and the first value information.

[0163] In some embodiments, the circumferential image features and the surrounding image features of pure vision are fused with point cloud features to improve the accuracy of vehicle as environment detection. That is, the circumferential image features, the surrounding image features and the point cloud features can be stitched together to obtain the first key information and the first value information.

[0164] In some embodiments, the two-dimensional panoramic features, the two-dimensional surrounding features, and the point cloud features are stitched together to determine the first key information corresponding to the detection task; the two-dimensional panoramic features, the three-dimensional panoramic position features, the two-dimensional surrounding features, the three-dimensional surrounding position features, and the point cloud features are stitched together to determine the first value information corresponding to the detection task.

[0165] In this embodiment of the present disclosure, feature extraction is performed on the radar point cloud data of the vehicle to obtain point cloud features. Based on the perimeter image features, the ring image features, and the point cloud features, first key information and first value information are determined so that the first key information and first value information include the purely visual perimeter image features, the ring image features, and the point cloud features, thereby improving the accuracy of detecting the vehicle's surrounding environment.

[0166] The following describes the application of the embodiments of this disclosure in real-world scenarios.

[0167] With the rapid development of autonomous driving technology, perception methods based on multi-source, multi-modal BEV fusion have become the mainstream technology for driving and parking perception, and are widely favored by the industry. BEV perception utilizes multi-source, multi-modal information from cameras and radar around the vehicle. By mapping the feature information of each sensor to the BEV space, it achieves omnidirectional feature information fusion and coverage from all directions (front, rear, left, right, far, medium, and near), thereby significantly improving the perception capability of important external targets and the overall road topology. This perception method, unlike single-view perception, directly performs spatial coverage information fusion of different sources and modalities at the feature layer of each sensor data. This not only avoids target interference from single sensors and the loss of useful information caused by post-fusion, but also greatly enhances the spatial feature representation capability, thereby effectively improving the accuracy of global environmental perception and detection.

[0168] In multimodal BEV perception, LiDAR can accurately reflect the 3D spatial depth and position of objects and their environment, offering unparalleled advantages in ranging and precise positioning of spatial objects compared to vision. For example, it has superior detection capabilities compared to vision for obstacle avoidance against sudden intrusions or unknown obstacles not on the whitelist. However, the persistently high cost of LiDAR significantly limits the widespread adoption of BEV perception methods based on laser-based image fusion, currently confining it to higher-end models and hindering its large-scale penetration into the mid-to-low-end vehicle market. Therefore, low-cost sensor fusion solutions, represented by pure vision, are more cost-effective for most companies in the industry, offering the possibility of large-scale market penetration. Visual image information is rich in detail and features, sufficient to fully reflect the diverse information of the vehicle's surrounding environment and the target's color, shape, texture, and pose, and has always been considered a superior data source for classification, recognition, and scene semantic representation. However, the drawbacks of relying solely on visual image content are also prominent: it only reflects 2D planar information, unlike laser point clouds which naturally possess good depth attributes, making it difficult to achieve high-precision measurement and positioning. Despite the fact that the academic community has proposed various methods using deep neural networks to improve the prediction accuracy of visual depth, the current depth prediction for images is still coarse-grained. How to achieve accurate localization of targets in 2D images in the 3D physical world remains a major challenge in this research field.

[0169] To address the aforementioned issues, without considering LiDAR sensors, an intuitive approach is to fully utilize image information from various onboard perspectives to enhance the comprehensive perception and localization capabilities of targets and the environment. Under the premise of supporting driving and parking functions, autonomous vehicles are equipped with panoramic and surround-view cameras sufficient to cover a 360° field of view around the vehicle, enabling detection coverage at close to medium distances. However, for mass production applications, due to significant differences between panoramic and surround-view images in terms of field of view (FOV) and distortion patterns, panoramic and surround-view images are generally processed independently for driving and parking tasks. Achieving true feature fusion at the feature level presents certain technical challenges, typically requiring the fusion of their output information during post-processing. This approach introduces two problems: firstly, two independent models increase the consumption of limited onboard resources and cause high latency due to contention; secondly, it cannot effectively compensate for the inherent blind spots of panoramic images. Therefore, an effective integrated network structure can be constructed to fully and efficiently integrate panoramic and all-around image information to form a true multi-view, multi-source perception input and output. This can not only significantly solve the aforementioned problems, but also improve the accuracy of perception and positioning.

[0170] Extensive research and practice in the fields of natural language processing and computer vision have demonstrated that, compared to CNN structures with limited visual receptive fields, neural network architectures based on attention mechanisms (Transformer network architecture) are superior in constructing associations between global and local features, and between homologous and non-homologous tasks, significantly improving the predictive ability for complex scenes. Based on this characteristic, this study fully leverages the advantages of the Transformer architecture, using the attention mechanism and efficient, low-overhead sparse query representation to establish abstract relationships between predicted objects and panoramic and surrounding image information, dynamic targets and static road structure information, and current moment and historical information in the BEV perception network. This achieves full association computation across all camera views and across all task content, helping to significantly improve the model's perception and prediction accuracy for complex environments while ensuring operational efficiency, demonstrating good feasibility for implementation.

[0171] The multi-view BEV perception task under the pinhole camera model only outputs 3D target information, cannot realize the fusion and utilization of panoramic fisheye camera images, does not support multi-task output such as road information, and does not consider the relationship between dynamic and static targets, thus limiting its application scenarios.

[0172] Furthermore, for multiple images acquired by multiple fisheye cameras, feature extraction is performed on the current frame fisheye image and historical frame fisheye images to obtain a first feature map corresponding to the current frame fisheye image and a second feature map corresponding to the historical frame fisheye images. Then, BEV merging features are obtained based on the first and second feature maps, and target tracking and detection are performed based on the BEV merging features. This method only performs feature extraction and mapping processing on fisheye images. Fisheye distortion inevitably has a significant negative impact on recognition and accurate positioning. In addition, the detection distance of fisheye surround view is limited. Furthermore, the model cannot reuse images from conventional pinhole cameras, and its application scenarios are also very limited.

[0173] This method employs a sparse query design based on the Transformer architecture, a time mechanism centered on dynamic target objects, and online model execution. Long-term historical cache information is propagated through frame-by-frame object queries. A motion perception layer normalization is also introduced to model object motion. Although this method builds a sparse BEV perception architecture based on Transformer, it only considers traditional panoramic image input and lacks a mechanism for fusing surround-view fisheye image information. The output task only includes specific 3D dynamic targets and lacks the ability to perceive static road structures. Consequently, it cannot construct the relationships between important environmental elements such as panoramic, surround-view, dynamic, and static views, making it difficult to handle complex multi-task scenarios.

[0174] Based on the above description, this disclosure provides a BEV (Battery Electric Vehicle) surround-view dynamic and static multi-task perception method using a sparse HAS-Propagation Transformer (Hierarchical Attention Structure Propagation Transformer) architecture. By constructing a Transformer model architecture based on temporally sparse feature representation, it achieves integrated perception and multi-task processing output of surround-view and panoramic images covering a 360° range around the vehicle. This Transformer model architecture relies on image information input from the vehicle's surround-view vision sensors to solve the problem of perceiving dynamic and static targets and the environment during driving and parking, reducing the cost of detection sensors. The integrated input-output structure of surround-view and panoramic views, on the one hand, avoids resource contention and delays caused by the independent operation of traditional surround-view and panoramic perception tasks, effectively improving information processing efficiency; on the other hand, by fusing and utilizing surround-view and panoramic information, it significantly improves perception accuracy. The Transformer architecture employs a multi-task sparse feature representation for dynamic and static perception. This not only effectively mitigates the significant memory consumption and latency degradation caused by the currently widely used intensive BEV feature mapping generation, but also, through the Hybrid Attention and AS Attention mechanisms in the HAS-Propagation Transformer structure, achieves abstract feature associations between dynamic targets and static road structures, between passable road areas and roadside lane lines, and between current and historical states. This results in a more accurate positional topology relationship between the output dynamic target and the static road environment. In summary, the embodiments disclosed in this disclosure fully utilize low-cost vehicle vision sensor information and construct effective temporal dynamic and static target associations based on the Transformer to achieve integrated and efficient perception output.

[0175] Figure 2 shows a schematic diagram of a vehicle environment detection structure using a sparse HAS-Propagation Transformer architecture proposed in an embodiment of this disclosure. The structure shown in Figure 2 includes an inference unit 200 and a training unit 240. The inference unit 200 comprises the following components: a panoramic pinhole camera imaging unit 201 acquires a panoramic image; a panoramic coding network 2D image feature encoding unit 202 performs 2D encoding of the panoramic image; and a panoramic imaging 3D pseudo-point cloud unit 203 and a panoramic 3D position encoding unit 204 complete 3D encoding of the panoramic image. A surround-view fisheye camera imaging unit 205 acquires a surround-view image; a surround-view coding network 2D image encoding unit 206 performs 2D encoding of the surround-view image; and a surround-view imaging 3D pseudo-point cloud unit 207 and a surround-view 3D position encoding unit 207 complete 3D encoding of the surround-view image. Unit 208 completes the 3D encoding of the surround view image; based on the vehicle positioning attitude and timestamp information 214, it caches the information of the historical moment in the historical cache information unit 212, and generates the historical embedding reconstruction vector in the historical embedding reconstruction generation unit 211 based on the historical cache information; it initializes the current query in the current query initialization unit 213 using the historical embedding reconstruction vector; it inputs the 2D encoded features of the surround view image, the 3D encoded features of the surround view image, the historical embedding reconstruction vector, and the current query initialization obtained by the above units into the HAS-Propagation Transformer network unit 220, and decodes the features output by the HAS-Propagation Transformer network unit 220 in the 3D object detection task decoding unit 231, the road segmentation detection task decoding unit 232, and the lane line detection task decoding unit 233, respectively. The training unit 240 consists of the following units: the current query initialization unit 213 is trained using the initial query optimization iteration 241; the 3D target detection task decoding unit 231 is trained using the 3D target annotation ground truth information 242; and the road segmentation detection task decoding unit 232 and the lane line detection task decoding unit 233 are trained using the road structure annotation ground truth information 243.

[0176] Figure 3 shows a schematic diagram of a network structure for vehicle environment detection using a sparse HAS-Propagation Transformer architecture proposed in an embodiment of this disclosure. As shown in Figure 3, the perimeter coding network 301 is used to process the perimeter images 1 to N of the current frame. a Perform 2D encoding to obtain panoramic features The panoramic 3D image is 3D encoded using panoramic 3D position coding 302 to obtain the panoramic 3D code. Using the look-around coding network 303 to view the look-around images 1 to N of the current frame sPerform 2D encoding to obtain the look-around features. The surround view image is 3D encoded using surround view 3D position coding 304 to obtain the surround view 3D code. Initialize the current frame initialization query information 308 based on the previous frame's embedding reconstruction vector 307 in the historical cache queue 305; align the information in the historical cache queue 305 with the current frame's spatiotemporal context and reconstruct it to obtain the historical frame embedding reconstruction vector 306; then, use the panoramic features... Panoramic 3D Coding Surround view features Surround view 3D coding The current frame initialization query information 308 and the historical frame embedding reconstruction vector 306 are input into the HAS-Propagation Transformer network unit 220, which outputs 3D object detection task query 309, road segmentation detection task query 310, and lane line detection task query 311. The 3D object detection decoding task header 312, road segmentation detection decoding task header 313, and lane line detection decoding task header 314 are used to decode the 3D object detection task query 309, road segmentation detection task query 310, and lane line detection task query 311, respectively, to obtain the current frame Top-K buffers 315. The current frame Top-K buffers 315 are stored in the historical buffer queue 305.

[0177] The implementation process of the BEV surround-view dynamic and static multi-task perception method of the sparse HAS-Propagation Transformer architecture in this embodiment is shown in Figure 4, and may include steps S401 to S406:

[0178] Step S401: Panoramic and 3D positional images: 2D feature encoding and 3D positional encoding;

[0179] The coordinate system provided in this embodiment is shown in FIG5, including camera coordinate system 501 and vehicle coordinate system 502.

[0180] First, feature encoding is performed on panoramic pinhole camera images and 360-degree fisheye camera images using two independent 2D convolutional coding backbones to obtain 2D encoded features of the panoramic images. and 2D encoded features of the panoramic image Then, using the intrinsic parameters and distortion parameter models of the panoramic camera and the circumferential camera respectively, distortion-free frustum pseudo-point clouds of each camera image with a certain depth interval are constructed. The coordinates of these pseudo-point clouds are then transformed from the camera coordinate system 501 to the vehicle coordinate system 502 using the corresponding camera extrinsic parameters, thus completing the 3D position encoding of each pixel in the panoramic image. 3D position encoding of each pixel in the panoramic image

[0181] In some embodiments, step S401 may include steps S4011 and S4012:

[0182] Step S4011: Perform 2D feature encoding on the panoramic camera image and the surround camera image. The backbone network can typically adopt, but is not limited to, common 2D convolutional network structures with a certain depth, such as ResNet, EfficientNet, SwinTransformer, and VoVNetV2, and all perimeter images and all toroidal images share their own set of network weights.

[0183] Step S4012: Perform 3D position encoding on the panoramic camera images and the surround camera images. Its operation method is as follows: for a camera image from a certain perspective:

[0184] First, based on the distortion model and distortion parameters of the panoramic or 360-degree camera, combined with the corresponding camera intrinsic parameters I... c ∈R 3×3 Calculate the true distortion vector direction of each pixel in the restored image in the camera coordinate system.

[0185] Next, based on the required generated BEV probe physical space size H B ×W B Set reasonable equal-interval depth segments {z k |k∈n}, and combined with the true vector direction of each pixel, construct the coordinate values ​​of each pixel (i,j) in the image in different depth segments in the camera coordinate system.

[0186] Then, using the extrinsic parameters of the corresponding camera relative to the vehicle's coordinate system, rotate R... c ∈R 3×3 and displacement T c ∈R 3×1 The information transforms the coordinates of each pixel in different depth segments to the vehicle's coordinate system.

[0187] Then, RefPE encoding is used to encode the coordinates of different depth segments corresponding to each pixel in the vehicle coordinate system. The encoding method is shown in formulas (1) and (2):

[0188] in, ξ represents the Fourier positional code, s i,j,k The weighted weight of depth segment k corresponding to pixel (i,j) can be set manually or obtained through training. Concat and MLP represent the stitching operation and multilayer perceptron network, respectively.

[0189] Step S402: Reconstruct and generate historical cache information embedding;

[0190] In some embodiments, the cached information of dynamic and static task representations for tasks such as 3D target detection (corresponding to the moving target detection task in the aforementioned embodiments), road passable area segmentation detection (corresponding to the road segmentation detection task in the aforementioned embodiments), and lane line edge detection (corresponding to the lane line detection task in the aforementioned embodiments) stored in the historical cache memory at frames t-1, t-2 to tn is read. Combined with the pose transformation relationship between each historical frame time (t-1, t-2...tn) and the current frame time t, the representations of each detection result are reconstructed to generate a corresponding high-dimensional reconstruction embedding vector spatially aligned with the current frame time t.

[0191] In some embodiments, the time periods t-1, t-2 to tn in the historical cache memory are shown in Figure 6. The FIFO cache 600 caches information from time periods t-1, t-2 to tn. The cached information 601 for frame t-1 includes global information 604 for frame t-1 and TopK. d Information on 3D object detection task in frame t-1 (605, TopK) s Road segmentation detection task information for frame t-1: 606, TopK l The t-1 frame lane edge detection task information 607 includes t-1 frame global information 604, which includes t-1 frame pose information 602 and t-1 frame timestamp information 603; the tk frame buffer information 611 includes tk frame global information 614 and TopK... d 3D object detection task information for each TK frame: 615, TopK s Road segmentation and detection task information for each tk frame: 616, TopK lThe lane line detection task information 617 for each tk frame includes tk frame global information 614, which includes tk frame pose information 612 and tk frame timestamp information 613. Optionally, taking time tk as an example, the information stored for that time includes: the vehicle's pose E relative to the global coordinate system at time tk. t-k and timestamp T t-k TopK d Context Embedding vectors for each 3D object detection task Central position Target speed TopK s The context embedding vector corresponding to each road segmentation detection task Central position TopK l The context embedding vector corresponding to the lane line detection task Central position Typically, considering that the dimension of the embedding vectors for various task contexts is C, the dimensions of the 3D target center position and velocity are 3 and 2 respectively, the dimension of the road and lane line center position is 2, and the dimensions of the timestamp and vehicle pose information at that moment are 3×3+3×1+1. The total cache memory usage is N×(13+K). d ×(C+5)+K s ×(C+2)+K l ×(C+2)).

[0192] In some embodiments, the pose transformation relationship between each historical frame time (t-1, t-2...tn) and the current frame time t is combined to reconstruct the information representation of each task. Taking historical time tk as an example, a high-dimensional reconstruction embedding vector corresponding to each detection task at time tk is generated that is spatially aligned with the current frame time t, as shown in Figures 7, 8, and 9. The operation method is as follows:

[0193] First, calculate the time difference ΔT = T between time tk and time t. t -T t-k And the relationship between the vehicle's position and pose at time tk and time t. And according to The center positions of the 3D object detection task, road segmentation detection task, and lane line detection task stored at time tk are transformed and aligned to the current vehicle coordinate system. Since the dimensions of the road and lane line center positions are 2, the height value of the ground in the vehicle coordinate system needs to be added in the height Z-axis direction to expand it to 3D, and then calculated according to the transformation formula.

[0194] Then, using the vehicle's position and pose transformation relationship Time difference ΔT and target velocity V (t-k) Based on the motion-aware regularization (MLN) method, the corresponding MLN coefficients (γ, β) are calculated as follows:

[0195] Where, ξ γ and ξ β These represent two independent linear transformation layers, with network weights obtained through learning. The target's motion velocity is relative to the 3D target. For the target's speed V along the road and lane lines (t-k) =0, meaning that the targets in the passable road area and the lane line targets have the same coefficient (γ,β), while for 3D moving targets, the coefficients (γ,β) are different because they have different speeds.

[0196] Then, combining the previously obtained MLN coefficients (γ,β), the task information cached at time tk is reconstructed by aligning it with the embedding at time t. The calculation method is as follows:

[0197] Here, MLP represents a multilayer perceptron network, and LN represents layer regularization. Since the physical objects corresponding to road segmentation detection are the center positions of various cube-shaped patches on the ground plane of the BEV space, it directly utilizes... right After conversion Generally, the various cube-shaped surfaces in the BEV space under the current vehicle coordinate system at time t are difficult to overlap and correspond to. Therefore, the nearest surface principle can be adopted, which means that the closest surface to the current vehicle is the surface that is closest to the current vehicle. The center position of the nearest piece of the face is used as The value is then substituted into the Embedding reconstruction calculation.

[0198] Step S403: Current frame Query construction and initialization;

[0199] In some embodiments, the Embedding vector is reconstructed based on the previous frame t-1 closest to the current frame t, and propagated to initialize the query information of the current frame. At the same time, it is combined with random initialization of some spatial locations of the current frame to complete the initial query construction for various dynamic and static tasks, including 3D targets, passable road areas, lane lines, etc. The feature vector content includes the context Embedding and location Embedding corresponding to all queries.

[0200] In some embodiments, initializing the current frame Query may include constructing the Query corresponding to each dynamic and static task, such as 3D object detection, road passable area segmentation, and lane line detection. The physical object meanings are shown in Figure 10. In the BEV space 1001, there are the vehicle center 1002, 3D object detection task Query 309, road segmentation detection task Query 310, and lane line detection task Query 311. 3D object detection task Query 309 corresponds to the initial position and context features of each candidate 3D object in the BEV space; road segmentation detection task Query 310 corresponds to the BEV ground plane S. H ×S W The context features corresponding to each rectangular patch of equal size are fixed at the center point of each patch; the lane line detection task Query311 corresponds to the initial geometric center position and context features of each lane line or road edge.

[0201] In some embodiments, the Query initialization information for each dynamic and static task in the current frame includes two parts. The first part is the propagation of the Embedding vector 1001 reconstructed at time t-1 of the previous frame to initialize the Query information 1002 of the current frame, that is: The second part is to randomly initialize a batch of queries at time t in the current frame, i.e.

[0202] In some embodiments, the first part initializes the TopK of the 3D target task in the current frame Query by the propagation of the Embedding vector reconstructed at time t-1 of the previous frame. d indivual TopK along the lane line task l indivual All are directly concatenated with the corresponding task query in the second part, while the TopK of the road task... s indivual In addition to being used in conjunction with the Query corresponding to the second part, it can also confirm the BEV space 2D ground plane S corresponding to these Embeddings reconstructed at time t-1 in the current frame's vehicle coordinate system. H ×S W The position of S in the rectangular block. H ×S W TopK in each road task query s It is initialized by propagation.

[0203] In some embodiments, the second part of the random initialization operation is as follows: the center position initialization of each 3D target task Query adopts the uniformly distributed coordinates in the BEV space 3D space under the vehicle coordinate system; the center position initialization of each road task Query adopts the S433 described in step S433. H ×S W The remaining S in the rectangular block after the previous frame's Propagation propagation H ×S W TopK s The center coordinates of each patch; the center position of each lane along the Query is the uniformly distributed coordinates of the position on the 2D ground plane of the BEV space in the vehicle coordinate system; the initial value of the context embedding vector of each task is set to 0 vector; to maintain feature consistency, the obtained Query position coordinate vectors of each task and the context initialization embedding vector are substituted into formulas (3) to (6) to obtain the second part of the initialization embedding, that is: Formula (3) can be set Substitute these values ​​into the equation to calculate the MLN coefficients (γ,β).

[0204] Step S404: HAS-Propagation Transformer calculation;

[0205] As shown in Figure 11, the information obtained in steps S401, S402, and S403 is used as the input to the HAS-Propagation Transformer network. This network consists of several layers of hybrid attention mechanism 1102 and panoramic / all-around attention mechanism 1101 (AS Attention) combined in series. The corresponding inputs are as follows: based on the embeddings of dynamic and static tasks at historical frames (t-1, t-2…tn) obtained in step S402 and the initialization of query information at the current frame t obtained in step S403, the two information are concatenated to form the input key vector and value vector of the Hybrid Attention. The Attention mechanism is used to establish the implicit feature relationships between dynamic targets and static road structures, between the passable road area and the roadside lane lines, and between the current state and historical states. Then, the output of the Hybrid Attention is used as the query input for the subsequent AS Attention, and the panoramic and all-around image 2D encoded features obtained in step S401 are used as the input. and 3D position encoding features The AS Attention mechanism forms the input key and value vectors, and then establishes feature associations between each dynamic and static task and the observed content of the panoramic and all-around images at time t in the current frame through the attention mechanism. In this way, after several layers of Hybrid Attention and AS Attention, the current frame Query completes the information association and update representation between itself and each dynamic and static task and with each historical time.

[0206] In some embodiments, the Hybrid Attention unit primarily constructs the relationships between each task at the current moment and its corresponding historical tasks, as well as between dynamic and static tasks. The AS Attention unit primarily constructs the relationships between dynamic and static tasks and the current panoramic and surrounding observations. Both Hybrid Attention and AS Attention employ the multi-head attention calculation method from the standard Transformer, as shown in the following formula:

[0207] Where Q, K, and V represent the inputs of the Attention unit: Query (corresponding to the query matrix in the aforementioned embodiments), Key (corresponding to the key matrix in the aforementioned embodiments), and Value (corresponding to the value matrix in the aforementioned embodiments), respectively. k This represents the dimension of the K vector. W O These are the weight coefficients of the multi-head attention mechanism.

[0208] In some embodiments, the input and output of the Hybrid Attention unit are as shown in Figure 11, and may include the following operations:

[0209] The query is set to propagate the Embedding vector Propagation reconstructed from the previous frame t-1 to the current frame t. And a batch of random initializations at time t of the current frame Composed of multiple parts, denoted as

[0210] Then, set the key and value to be exactly the same, both being the high-dimensional reconstructed embedding vector of the current frame at time t, which is the remaining time (t-2...tn) after removing time t-1 from the historical cache. The current initialization constructed above Together they are assembled to form, that is and

[0211] Then, the constructed Query, Key, and Value are substituted into the multi-head Attention calculation formula (7) to obtain the output of the Hybrid Attention unit. It is used as input for the subsequent AS Attention unit.

[0212] In some embodiments, the input and output of the AS Attention unit are as shown in Figure 11, and may include the following operations:

[0213] The query is set to the output of the Hybrid Attention unit, i.e.

[0214] Then, Value is set to the 2D encoded features of all the panoramic images in step S401. and 2D encoded features of the panoramic image splicing and combining, that is

[0215] Next, the Key is set to the 2D encoded features of all the panoramic images in step 301. and 3D position features Summarization and 2D encoding features of the panoramic image and 3D position features Adding together and combining them results in...

[0216] Then, the constructed Query, Key, and Value are substituted into the multi-head Attention calculation formula (7) to obtain the output of the AS Attention unit. If this layer is not the last layer It is used as input to the next layer of the HAS-Propagation Transformer network, such as the last layer. It is used for 3D object detection, road passable area segmentation, and lane line edge detection and decoding.

[0217] Step S405: Dynamic and static multi-task decoding;

[0218] In some embodiments, the 3D target query, road drivable area query, and lane line edge query output by the HAS-Propagation Transformer network obtained in step S404 are respectively connected to the corresponding decoding head network of each task to complete the multi-task output of dynamic and static target and environment perception; then, the decoding confidence of each task query is sorted in descending order from high to low, and the top K ranked queries are selected respectively. d 3D target, TopKs Road segmentation and TopK l For each lane, the corresponding decoding information is used to update the historical cache memory in a first-in-first-out (FIFO) manner for use in the next frame.

[0219] In some embodiments, the 3D target query, road drivable area query, and lane line edge query output by the HAS-Propagation Transformer are decoded respectively. Specifically: the decoding of the 3D target query can employ a simple multilayer perceptron network with an input dimension of C and an output dimension that includes information such as target confidence, target category, center coordinates, bounding box dimensions, target orientation, and target speed, which can be set according to actual needs; the decoding of the road drivable area query can segment pixel-level location regions on the BEV plane, since each road query corresponds to the BEV plane S. H ×S W One of a number of rectangular facets, each facet corresponding to r h ×r w Pixel size, therefore, an input dimension of C and an output dimension of r can be used. h ×r w A simple multilayer perceptron network with added 1-dimensional confidence; of course, a simple CNN network could also be used here to evaluate r. h ×r w Pixel segmentation is performed; the lane line edge query also uses a multilayer perceptron network, with an input dimension of C, but its output dimension depends on the vectorized representation of the lane line edge. Typical representations include MapTR, PETRv2, etc., or even the 3D lane line PersFormer method (if using 3D lane lines, the height coordinates of the initial center point of the query mentioned above can be adapted and adjusted). It can be weighed according to actual needs and complexity constraints, and no constraints are imposed here.

[0220] In some embodiments, the decoding confidence scores of the queries for each task—3D object detection, road passable area segmentation, and lane line edge detection—are sorted in descending order from high to low, and the top-ranked TopK queries are selected. d 3D target, TopK s Road segmentation and TopK l The decoding information corresponding to each lane line is Query, and then the historical cache memory data information is stored according to each task. The historical cache memory is updated in the first-in-first-out (FIFO) manner for use in the next frame processing.

[0221] In some embodiments, the current frame lane line along the center position stored in the history cache memory The calculation method is as follows: First, extract the coordinates of all k key points {p1, p2, ..., p} after decoding the lane line or curb. k The C-dimensional lane line along the Query is taken as input, and a multilayer perceptron network is used to obtain the k-dimensional weights [w1, w2, ..., w] corresponding to each key point. k Then, the weighted geometric center of the lane line is calculated according to the following formula:

[0222] To maintain the effectiveness of the weight distribution in the multilayer perceptron network, the coordinates of the k key points {p1, p2, ..., p...} are substituted into formula (8). k The system will sort the data according to certain rules and logic beforehand based on the decoded information.

[0223] Step S406: Sample library preparation, loss function design, model training and inference.

[0224] In some embodiments, the BEV perception network model described in steps S401 to S405 is subjected to supervised training. 2D visual images from the vehicle's panoramic pinhole camera and surround-view fisheye camera at each time point are acquired to align and pair sensor timestamp data. Then, for different specific perception tasks, corresponding task data sample annotations are completed to generate ground truth values ​​required for supervised training of the network model. Subsequently, by constructing a corresponding multi-task loss function and combining it with an effective strategy to accelerate training convergence, the generalization training of the BEV perception network model is completed to obtain robust network weights. The trained network weights are then provided to the network model for inference.

[0225] In some embodiments, when aligning and pairing the 2D visual images of the vehicle-mounted panoramic pinhole camera and the surround fisheye camera, if the exposure frequencies are the same, hardware synchronous triggering alignment can be considered. However, if the exposure frequencies are different, nearest neighbor time pairing can also be used. Generally, visual sensors have a high sampling frequency, so the time delay error introduced by nearest neighbor pairing is generally controllable.

[0226] In some embodiments, the total loss function L for model training is a weighted sum of the prediction losses from all perceptual tasks, where the 3D object detection loss L... det Generally, the loss can be obtained by weighting the classification Focal loss and the 3D bounding box regression L1 loss, while the road passable area segmentation loss L... segGenerally, the loss can be obtained by cross-entropy calculation, while the lane line detection loss can be obtained by calculating the position deviation of key points and geometric center using L1 loss. Random data augmentation methods used to improve the generalization of network training include, but are not limited to, random scale transformation, region clipping, symmetric mirroring, color transformation, and mesh masking.

[0227] Based on the foregoing embodiments, this disclosure provides a vehicle environment detection device, as shown in FIG12. The vehicle environment detection device 1200 includes: a feature extraction unit 1201 configured to extract features from a circumferential view image and a surround view image of the vehicle, respectively, to obtain circumferential view image features and surround view image features; a first determination unit 1202 configured to determine first query information based on the task type corresponding to the detection task; a second determination unit 1203 configured to determine first key information and first value information based on the circumferential view image features and the surround view image features; a processing unit 1204 configured to process the first query information, the first key information, and the first value information using an attention mechanism to obtain target fusion features; and a third determination unit 1205 configured to detect the surrounding environment of the vehicle based on the target fusion features and determine the detection result corresponding to the detection task.

[0228] In some embodiments, the panoramic image features include two-dimensional panoramic features and three-dimensional panoramic position features, and the ring view image features include two-dimensional ring view features and three-dimensional ring view position features; the feature extraction unit includes: a first encoding device configured to perform two-dimensional encoding on the panoramic image and the ring view image to determine the two-dimensional panoramic features and the two-dimensional ring view features; and a second encoding device configured to perform three-dimensional encoding on the panoramic image and the ring view image to determine the three-dimensional panoramic position features and the three-dimensional ring view position features.

[0229] In some embodiments, the first determining unit includes: a first determining device configured to determine second query information, second key information, and second value information corresponding to the detection task based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical time; and a processing device configured to process the second query information, the second key information, and the second value information using the attention mechanism to obtain the first query information.

[0230] In some embodiments, the first determining device includes: a first determining sub-device configured to reconstruct the detection results corresponding to the detection tasks at each of the historical times based on the pose transformation relationship of the vehicle at each of the historical times and the current time, and determine the reconstruction features corresponding to the detection tasks at each of the historical times; a second determining sub-device configured to determine the initial reconstruction features corresponding to the detection task at the current time based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical time; the first historical time is the time before the current time; a third determining sub-device configured to determine the initial reconstruction features corresponding to the detection task at the current time as the second query information; and a fourth determining sub-device configured to determine the second key information and the second value information based on the reconstruction features corresponding to the detection tasks at each of the second historical times and the initial reconstruction features corresponding to the detection task at the current time; the second historical times are historical times other than the first historical time.

[0231] In some embodiments, the first determining sub-device is further configured to: align the task information corresponding to the detection task at each of the historical moments to the current moment based on the pose transformation relationship, and determine the updated detection result corresponding to the detection task at each of the historical moments; and reconstruct the updated detection result corresponding to the detection task at each of the historical moments based on the pose transformation relationship, the time difference between each of the historical moments and the current moment, and the motion speed corresponding to the detection task, and determine the reconstruction feature corresponding to the detection task at each of the historical moments.

[0232] In some embodiments, the second determining sub-device is further configured to: determine a first initial reconstruction feature corresponding to the detection task at the current time based on the reconstruction feature corresponding to the detection task at the first historical time; randomly generate a second initial reconstruction feature corresponding to the detection task at the current time based on the task type corresponding to the detection task; and concatenate the first initial reconstruction feature and the second initial reconstruction feature to determine the initial reconstruction feature corresponding to the current time.

[0233] In some embodiments, the second determining sub-device is further configured to: randomly generate an initial feature corresponding to the detection task at the current time based on the task type corresponding to the detection task; reconstruct the initial feature corresponding to the detection task to determine a second initial reconstructed feature corresponding to the detection task at the current time.

[0234] In some embodiments, the second determining sub-device is further configured to: when the detection task includes a moving target detection task, uniformly sample coordinates in three-dimensional space to determine initial features corresponding to the moving target detection task; when the detection task includes a road segmentation detection task, determine initial features corresponding to the road segmentation detection task based on the coordinates of the center positions of each sub-region within a set vehicle-accessible area; when the detection task includes a lane line detection task, uniformly sample coordinates in two-dimensional space to determine initial features corresponding to the lane line detection task.

[0235] In some embodiments, the second determining unit includes: a second determining device configured to stitch together the two-dimensional panoramic feature and the two-dimensional surround view feature to determine first key information corresponding to the detection task; and a stitching device configured to stitch together the two-dimensional panoramic feature, the three-dimensional panoramic position feature, the two-dimensional surround view feature, and the three-dimensional surround view position feature to determine first value information corresponding to the detection task.

[0236] In some embodiments, the processing unit includes: a generation device configured to generate an attention matrix by using the first query information as a query matrix of the attention mechanism and the first key information as a key matrix of the attention mechanism; and a third determination device configured to determine the target fusion feature based on the attention matrix and the first value information.

[0237] In some embodiments, the first determining unit includes: a fourth determining device configured to determine query information corresponding to each detection task based on the task type corresponding to the detection task; and a fifth determining device configured to concatenate the query information corresponding to each detection task to determine the first query information.

[0238] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or structures of the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0239] This disclosure also proposes a computer device including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the instructions to implement some or all of the steps in the above-described method.

[0240] This disclosure also proposes a computer-readable storage medium storing computer instructions that, when executed by a processor, implement some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0241] This disclosure also proposes a computer program including computer-readable code, wherein, when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0242] This disclosure also proposes a computer program product comprising a non-transitory computer-readable storage medium storing computer instructions that, when read and executed by a computer, implement some or all of the steps in the above-described method. This computer program product can be implemented in hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium; in other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0243] The foregoing descriptions of the various embodiments tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the device, storage medium, computer program, and computer program product embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0244] This disclosure provides a hardware entity for a computer device, as shown in FIG13. The hardware entity of the computer device 1300 includes: a processor 1301 that generally controls the overall operation of the computer device 1300; a communication interface 1302 that enables the computer device to communicate with other terminals or servers via a network; and a memory 1303 configured to store instructions and applications executable by the processor 1301, and may also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1301 and various parts of the computer device 1300, which may be implemented using flash memory or random access memory (RAM). Data transfer between the processor 1301, the communication interface 1302, and the memory 1303 may be performed via a bus 1304.

[0245] Optionally, the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the embodiments of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Optionally, in the various embodiments of the embodiments of this disclosure, the sequence number of the above steps / processes does not imply the order of execution; the execution order of each step / process can be determined based on its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0246] In this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0247] In the embodiments disclosed herein, the devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0248] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected to implement the embodiments of this disclosure according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in hardware or in a combination of hardware and software functional units.

[0249] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by instruction-related hardware. The aforementioned instructions can be stored in a computer-readable storage medium. When the instructions are executed, they perform the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0250] Alternatively, when the integrated units of the embodiments of this disclosure are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0251] The above embodiments are merely illustrative of the embodiments of this disclosure, and the protection scope of the embodiments of this disclosure is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the embodiments of this disclosure are all within the protection scope of the embodiments of this disclosure.

Claims

1. A method for environmental testing of a vehicle, comprising: Feature extraction was performed on the circumferential and 360-degree views of the vehicle to obtain the circumferential view features and 360-degree view features respectively. Based on the task type corresponding to the detection task, determine the first query information; Based on the perimeter image features and the annular image features, determine the first key information and the first value information; By using an attention mechanism, the first query information, the first key information, and the first value information are processed to obtain the target fusion feature; and Based on the target fusion features, the surrounding environment of the vehicle is detected, and the detection result corresponding to the detection task is determined.

2. The method according to claim 1, wherein, The circumferential image features include two-dimensional circumferential features and three-dimensional circumferential position features, and the surround image features include two-dimensional surround features and three-dimensional surround position features; The step of extracting features from the circumferential and surround views of the vehicle to obtain circumferential view features and surround view features respectively includes: Two-dimensional encoding is performed on the perimeter view image and the ring view image to determine the two-dimensional perimeter view feature and the two-dimensional ring view feature; and The 3D circumferential view image and the 3D surround view image are 3D encoded to determine the 3D circumferential view position features and the 3D surround view position features.

3. The method according to claim 1, wherein, The determination of the first query information based on the task type corresponding to the detection task includes: Based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical time, determine the second query information, second key information, and second value information corresponding to the detection task; and The attention mechanism is used to process the second query information, the second key information, and the second value information to obtain the first query information.

4. The method according to claim 3, wherein, The step of determining the second query information, second key information, and second value information corresponding to the detection task based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical time includes: Based on the pose transformation relationship of the vehicle at each historical moment and the current moment, the detection results corresponding to the detection task at each historical moment are reconstructed to determine the reconstruction features corresponding to the detection task at each historical moment. Based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment, the initial reconstruction features corresponding to the detection task at the current moment are determined; wherein, the first historical moment is the previous moment of the current moment; The initial reconstruction feature corresponding to the detection task at the current moment is determined as the second query information; and Based on the reconstruction features corresponding to the detection task at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment, the second key information and the second value information are determined; the second historical moment is a historical moment other than the first historical moment.

5. The method according to claim 4, wherein, The method involves reconstructing the detection results corresponding to the detection task at each historical moment based on the pose transformation relationship between the vehicle at each historical moment and the current moment, and determining the reconstruction features corresponding to the detection task at each historical moment, including: Based on the pose transformation relationship, the detection results corresponding to the detection tasks at each of the historical time moments are aligned to the current time moment to determine the updated detection results corresponding to the detection tasks at each of the historical time moments; and Based on the pose transformation relationship, the time difference between each historical moment and the current moment, and the motion speed corresponding to the detection task, the updated detection results corresponding to the detection task at each historical moment are reconstructed to determine the reconstructed features corresponding to the detection task at each historical moment.

6. The method according to claim 4, wherein, The step of determining the initial reconstruction features corresponding to the detection task at the current time based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical time includes: Based on the reconstruction features corresponding to the detection task at the first historical moment, the first initial reconstruction features corresponding to the detection task at the current moment are determined. Based on the task type corresponding to the detection task, a second initial reconstruction feature corresponding to the detection task at the current time is randomly generated; and The first initial reconstruction feature and the second initial reconstruction feature are concatenated to determine the initial reconstruction feature corresponding to the current time.

7. The method according to claim 6, wherein, The step of randomly generating a second initial reconstruction feature corresponding to the detection task at the current time based on the task type corresponding to the detection task includes: Based on the task type corresponding to the detection task, an initial feature corresponding to the detection task at the current time is randomly generated; and The initial features corresponding to the detection task are reconstructed to determine the second initial reconstructed features corresponding to the detection task at the current time.

8. The method according to claim 7, wherein, The initial feature corresponding to the detection task at the current moment is randomly generated based on the task type corresponding to the detection task, including at least one of the following: When the detection task includes a moving target detection task, the coordinates in the three-dimensional space are uniformly sampled to determine the initial features corresponding to the moving target detection task; When the detection task includes a road segmentation detection task, the initial feature corresponding to the road segmentation detection task is determined based on the coordinates of the center positions of each sub-region in the set vehicle passable area. and When the detection task includes a lane detection task, the coordinates in the two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane detection task.

9. The method according to claim 2, wherein, The determination of the first key information and the first value information based on the perimeter image features and the ring image features includes: The two-dimensional panoramic features and the two-dimensional all-around features are concatenated to determine the first key information corresponding to the detection task; and The two-dimensional panoramic feature, the three-dimensional panoramic position feature, the two-dimensional surround view feature, and the three-dimensional surround view position feature are concatenated to determine the first value information corresponding to the detection task.

10. The method according to any one of claims 1 to 9, wherein, The process of using an attention mechanism to process the first query information, the first key information, and the first value information to obtain target fusion features includes: Using the first query information as the query matrix of the attention mechanism and the first key information as the key matrix of the attention mechanism, an attention matrix is ​​generated; and Based on the attention matrix and the first value information, the target fusion feature is determined.

11. The method according to any one of claims 1 to 9, wherein, The number of detection tasks is multiple; the determination of the first query information based on the task type corresponding to the detection task includes: For each detection task, based on the task type corresponding to the detection task, determine the query information corresponding to the detection task; and The query information corresponding to each of the detection tasks is concatenated to determine the first query information.

12. The method according to any one of claims 1 to 9, further comprising: Feature extraction is performed on the radar point cloud data of the vehicle to obtain point cloud features; The determination of the first key information and the first value information based on the perimeter image features and the ring image features includes: Based on the perimeter image features, the toroidal image features, and the point cloud features, the first key information and the first value information are determined.

13. An environmental detection device for a vehicle, comprising: The feature extraction unit is configured to extract features from the circumferential view image and the surround view image of the vehicle, respectively, to obtain the circumferential view image features and the surround view image features; The first determining unit is configured to determine the first query information based on the task type corresponding to the detection task; The second determining unit is configured to determine first key information and first value information based on the perimeter image features and the ring image features; The processing unit is configured to use an attention mechanism to process the first query information, the first key information, and the first value information to obtain target fusion features; and The third determining unit is configured to detect the surrounding environment of the vehicle based on the target fusion features and determine the detection result corresponding to the detection task.

14. A computer device comprising a memory and a processor, the memory storing computer instructions executable on the processor, the processor executing the instructions to implement the method of any one of claims 1 to 12.

15. A vehicle comprising the computer equipment as described in claim 14.

16. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method of any one of claims 1 to 12.

17. A computer program product comprising computer instructions that, when executed by a processor, implement the method of any one of claims 1 to 12.

Images