Camera-radar fusion for back-projection-based three-dimensional recognition

Abstract

A technology for camera-radar fusion for back-projection-based three-dimensional recognition is disclosed. A three-dimensional object detection method according to one embodiment may comprise the steps of: collecting an image from a camera; collecting radar points from a radar; projecting a cloud of the collected radar points onto an image coordinate system for the image in order to make the pixels of the collected image correspond to the collected radar points; generating a bird's-eye view (BEV) feature by converting the context feature of the image into a BEV space; integrating, into the BEV feature, context information extracted from the radar point cloud; and detecting or segmenting a three-dimensional object by using the BEV feature.

Description

Camera-Radar Fusion for Backprojection-Based 3D Perception

[0001] The following description relates to camera-radar fusion technology for back-projection-based 3D perception, and more specifically, to a new camera-radar fusion-based 3D object detection method and system.

[0002] Accurately perceiving the surroundings of an autonomous vehicle in three dimensions plays an important role in terms of safety, and the types and characteristics of the sensors used for this purpose are as follows.

[0003] Images obtained from a camera contain semantic and contextual information, but there are limitations in obtaining distance information to objects in three-dimensional space.

[0004] - Radar provides information on the distance and velocity of objects even in adverse weather conditions, but point clouds are sparse and altitude information cannot be determined.

[0005] While using LiDAR allows for accurate determination of object geometric information, it has the disadvantage of being significantly more expensive than sensors such as cameras or radar.

[0006] Due to the differing characteristics of these sensors, there is a need for methods to fuse and utilize the two sensors, and recently, camera-radar fusion technology, which is mutually complementary and low-cost, is gaining attention.

[0007] A novel camera-radar fusion-based 3D object detection method and system using backprojection to mitigate depth ambiguity is provided.

[0008] A method for detecting a three-dimensional object in a three-dimensional object detection system implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and the method comprises: a step of collecting an image from a camera by the at least one processor; a step of collecting radar points from a radar by the at least one processor; a step of projecting a cloud of the collected radar points onto an image coordinate system for the image by the at least one processor so as to correspond the pixels of the collected image with the collected radar points to each other; a step of generating BEV features by converting the context features of the image into a BEV (Bird's-Eye View) space by the at least one processor; a step of integrating context information extracted from the radar point cloud into the BEV features by the at least one processor; and a step of detecting or segmenting a three-dimensional object using the BEV features by the at least one processor.

[0009] According to one aspect, the step of generating the BEV feature may be characterized by generating the BEV feature map using an inverse projection-based view transformation for the image feature map of the image.

[0010] According to another aspect, the step of generating the BEV features may be characterized by generating the BEV feature map by integrating the contextual features of the image into the BEV feature map.

[0011] According to another aspect, the step of generating the BEV feature may be characterized by comprising: a step of calculating the outer product of radar occupancy information extracted from the cloud of the radar point and image occupancy information extracted from the image; a step of calculating an extended feature map using the outer product and the contextual features of the image; and a step of generating a first BEV feature encoded with depth-aware image features through a depth-aware spatial cross attention operation between the extended feature map and the BEV query.

[0012] According to another aspect, the radar occupancy information includes occupancy information in a frustum view generated by voxelizing the cloud of the radar points into a camera frustum view according to the CRN, and the depth perception spatial cross attention operation may be characterized by including a deformable attention operation using the extended feature map as a key, the camera parameters as a value, and the BEV query as a query.

[0013] According to another aspect, the step of generating the BEV feature may further include the step of generating a second BEV feature encoded with the contextual feature of the image using the depth distribution predicted in the image.

[0014] According to another aspect, the step of generating the second BEV feature may be characterized by generating the second BEV feature through a depth-aware spatial cross attention operation based on the depth distribution and image distance information of the image.

[0015] According to another aspect, the step of integrating the extracted context information into the BEV features may be characterized by integrating the radar context information into the BEV feature map through a context-aware spatial cross attention operation between the radar context information and the BEV features.

[0016] According to another aspect, the context-aware spatial cross-attention operation may be characterized by including a deformable attention operation that uses the context information of the radar as key and value and the BEV feature as a query.

[0017] According to another aspect, the context information as a key may include the Radar Cross Section (RCS) and Doppler velocity, and the context information as a value may include a matrix that projects the radar point onto a context feature map in a frustum view in the BEV.

[0018] According to another aspect, the steps of projecting the cloud of the collected radar points onto an image coordinate system for the image, generating the BEV features, and integrating the extracted contextual information into the BEV features may be characterized by being performed using a 3D object detection model trained using detection loss, depth loss, and occupancy loss.

[0019] According to another aspect, the detection loss may be characterized by including an L1 loss for bounding box regression and a focal loss for classification, the depth loss may include a binary cross-entropy loss to supervise the depth distribution, and the occupancy loss may include a Gaussian focal loss based on a heatmap obtained from a 2D bounding box.

[0020] A computer program stored on a computer-readable recording medium is provided to be combined with a computer device to execute the above method on the computer device.

[0021] A computer-readable recording medium is provided on which a program for executing the above method is recorded on a computer device.

[0022] A three-dimensional object detection system implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and wherein the at least one processor collects an image from a camera and collects radar points from a radar, projects a cloud of the collected radar points onto an image coordinate system for the image to correspond pixels of the collected image with the collected radar points, converts contextual features of the image into a BEV (Bird's-Eye View) space to generate BEV features, integrates contextual information extracted from the radar point cloud into the BEV features, and detects or segments a three-dimensional object using the BEV features.

[0023] A new camera-radar fusion-based 3D object detection method and system using backprojection to mitigate depth ambiguity can be provided.

[0024] FIG. 1 is a diagram illustrating an example of camera-radar fusion according to the prior art.

[0025] FIG. 2 is a drawing illustrating an example of CRAB according to an embodiment of the present invention.

[0026] FIG. 3 is a drawing illustrating an example of the overall structure of a CRAB in one embodiment of the present invention.

[0027] FIG. 4 is a diagram illustrating an example of radar occupancy information-based spatial cross attention according to an embodiment of the present invention.

[0028] FIG. 5 is a diagram illustrating a comparative example of encoded BEV features in an embodiment of the present invention.

[0029] FIG. 6 is a diagram illustrating an example of the qualitative result of three-dimensional object detection in one embodiment of the present invention.

[0030] FIG. 7 is a diagram illustrating an example of the qualitative results of BEV splitting in the nuScenes verification set in one embodiment of the present invention.

[0031] FIG. 8 is a diagram illustrating an example of a three-dimensional object detection method according to an embodiment of the present invention.

[0032] FIG. 9 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention.

[0033] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0034] A three-dimensional object detection method and system according to embodiments of the present invention may be implemented by at least one computer device. In this case, a computer program according to an embodiment of the present invention may be installed and run on at least one computer device, and the computer device may perform a three-dimensional object detection method according to embodiments of the present invention under the control of the run computer program. The above-described computer program may be stored on a computer-readable recording medium to be combined with at least one computer device to execute the three-dimensional object detection method on the computer.

[0035] Recently, camera-radar fusion-based 3D object detection methods have gained attention in Bird's Eye View (BEV) due to the complementary nature and cost-effectiveness of these sensors. Previous approaches using forward projection struggle with generating sparse BEV features, whereas methods using backward projection overlook depth ambiguity, leading to false positives.

[0036] In embodiments of the present invention, to address the aforementioned limitations, a camera-radar fusion-based 3D object detection (detection and / or segmentation) model (hereinafter referred to as 'CRAB (Camera-Radar fusion for reducing depth Ambiguity in Backward projection-based view transformation)') that utilizes back-projection to mitigate depth ambiguity may be provided, and a 3D object detection method and system using such CRAB may be provided. The CRAB may apply a method to reduce depth ambiguity through a back-projection technique utilizing radar. During view transformation, CRAB may aggregate perspective view image contextual features into BEV queries. This combines dense but unreliable depth distributions obtained from images with sparse but precise depth information obtained from radar occupancy information, thereby improving depth differentiation between queries on the same raycast. Additionally, CRAB may introduce a feature map containing radar contextual information and spatial cross-attention to enhance the understanding of the 3D scene. Evaluation results on the 'nuScenes' open dataset showed that the proposed approach achieved state-of-the-art performance among back-projection-based camera-radar fusion methods, recording 61.2% NDS and 53.1% mAP in 3D object detection.

[0037] 1. Introduction

[0038] Accurate perception of the surrounding 3D environment plays a crucial role in the fields of autonomous driving and mobile robotics. In these areas, cameras, LiDAR, and radar are the most commonly used sensors for 3D object detection. Due to their differing characteristics, the need for multi-sensor fusion technology is increasing. While research on camera-LiDAR fusion is active, it is vulnerable to performance degradation in adverse weather conditions, and the high cost of LiDAR poses a significant obstacle to the mass production of autonomous vehicles. Recently, research on fusing low-cost cameras and radar has become active, thanks to the complementary characteristics of the two sensors. In particular, images captured by cameras contain dense semantic and contextual information in the form of pixels in angular directions. However, cameras cannot accurately measure distance (depth) in 3D space because this is an inherently ill-posed problem. On the other hand, radar provides accurate distance information and possesses robust characteristics even in adverse weather environments. However, radar point clouds have limitations such as sparsity, noise caused by multipath effects, and limited angular resolution. Therefore, optimally fusing these complementary sensors can significantly improve performance by maximizing their respective strengths and minimizing their weaknesses.

[0039] Figure 1 illustrates an example of camera-radar fusion according to the prior art. CRAFT (Channel and Region Attention for 3D Fusion) shown in Figure 1 (a) directly generates 3D object candidates using detectors in a view image, filters out noisy radar points outside the candidate region during the fusion process, and applies 2D deformable attention. CRN (Camera-Radar Network) shown in Figure 1 (b) integrates radar occupancy information in the 'Lift' stage using orthographic projection, adaptively fuses BEV features of the image and radar, but uses 2D deformable attention without post-processing.

[0040] These existing fusion techniques generally follow a method of using 3D object candidates extracted from perspective view images and then filtering out noisy radar point clouds (e.g., Fig. 1(a)). Subsequently, fusion is achieved through operations such as attention or simple concatenation. However, these methods rely heavily on the performance of camera-based 3D object detection models, and spatial mismatch issues may arise when downstream tasks such as segmentation, path planning, and prediction are performed in BEV space. Meanwhile, recent camera-radar fusion technologies are primarily performed in BEV space due to their advantages, such as reduced object size variance, agnostic operation, and the ability to integrate coordinate systems from various sensors. These technologies mainly utilize the two existing view transformation methods: orthographic and back-projection. Orthographic-based approaches fail to address the problem of BEV features becoming sparser as the distance from the ego vehicle increases, and they have limitations in that they rely excessively on depth prediction without post-processing. On the other hand, techniques utilizing backprojection do not effectively solve the problem where BEV queries on the same ray have identical image features, resulting in a lack of depth differentiation.

[0041] CRAB according to the embodiments of the present invention can overcome the aforementioned limitations.

[0042] FIG. 2 illustrates an example of CRAB according to an embodiment of the present invention. CRAB is based on backprojection and uses radar occupancy information across multiple layers for 3D deformable attention, ensuring clear depth distinction while converting image context into BEVs. Subsequently, 2D deformable attention aggregates radar context features into BEV queries. This CRAB is a backprojection-based 3D object detection (detection and / or segmentation) model that can mitigate depth ambiguity by utilizing camera-radar fusion. Radar Occupancy-guided Spatial Cross Attention (ROSCA), a module that CRAB may include, performs more accurate depth estimation by additionally utilizing radar occupancy information in addition to the depth distribution used in the previous DFA3D (Deformable Attention for 3D) method, thereby generating precise BEV features. Furthermore, to effectively utilize radar context information, CRAB can include an RCSCA (Radar Context-aware Spatial Cross Attention) module. RCSCA enables deformable attention between frustum view-shaped radar contextual feature maps and BEV queries. Through this, even queries on the same ray can be projected to different depth locations, allowing for depth-distinguishing features to be obtained. Experimental results using the public 'nuScenes' dataset showed that CRAB achieved the highest performance among back-projection-based camera-radar fusion models. Additionally, the following effects of CRAB were verified through further experiments.

[0043] CRAB can mitigate depth ambiguity issues in back-projection-based view transformations through camera-radar fusion. CRAB can generate semantically rich and spatially accurate BEV features capable of understanding 3D scenes by introducing ROSCA and RCSCA modules.

[0044] As a result of conducting experiments on the public dataset 'nuScenes', CRAB recorded the best performance among back-projection-based 3D object detection camera-radar fusion models.

[0045] 2. Related Research

[0046] 2.1 Fusion in Perspective Space

[0047] Since view discrepancy exists between images and radar point clouds, it is required to fuse data obtained from the two sensors in an integrated space.

[0048] 2.1 Fusion in Perspective Space

[0049] GRIF Net (Gating Region of Interest Fusion Network) fuses Regions of Interest (RoIs) obtained from each modality through a gating mechanism. CenterFusion and CRAFT filter 3D object candidates obtained in view space by concatenating them with noisy radar points. CenterFusion uses simple concatenation in the channel direction, whereas CRAFT applies deformable attention for fusion. MVFusion (Multi-View Fusion) aligns semantic information from radar and image features and then performs fusion in the global space using cross-attention. However, these approaches are difficult to easily extend to other downstream tasks.

[0050] B. Convergence in BEV Space

[0051] Thanks to performance improvements in BEVs for camera-based 3D object detection, subsequent studies perform fusion in the BEV integration space. Both CRN and RCBEV (Radar Camera Bird's Eye View) are based on orthographic view transformation. CRN utilizes radar occupancy information during the 'Lift' process and resolves the spatial misalignment problem by adaptively aggregating BEV features from images and radar through deformable attention. RCBEV simply concatenates BEV features obtained from various modalities and then fuses the resulting image feature map heatmap with the radar heatmap using a CNN. However, neither method solves the problem of sparse BEV generation in orthographics. RCMFusion (Radar Camera Fusion with Multi-level Refinement) and TransCAR (Transformational Camera-Radar Fusion) are both based on backprojection. RCMFusion encodes BEVs by integrating radar guidance at the feature level and then performs refinement at the instance level. TransCAR uses a soft-association method between vision-updated queries and radar features instead of hard-association based on sensor calibration. However, both methods have a significant limitation in that they fail to resolve the depth ambiguity issue that occurs during the view transformation process.

[0052] 3. Examples of the present invention

[0053] CRAB can be composed of two modules that mitigate the depth ambiguity problem occurring in conventional technology and effectively fuse various modalities.

[0054] FIG. 3 is a diagram illustrating an example of the overall structure of a CRAB in an embodiment of the present invention. The CRAB can extract image and radar features from their respective backbones (Image Backbone, Radar Backbone). Then, BEV features obtained through Depth Distribution Guided Spatial Cross Attention and (a) Radar Occupancy Information Guided Spatial Cross Attention (ROSCA) can be fused together. The fused BEV features and radar context features can pass through (b) Radar Context Aware Spatial Cross Attention (RCSCA) and finally pass through a task-specific head. More specifically, as shown in FIG. 3, the CRAB may be composed of an image backbone and a radar backbone, Depth Distribution Guided Spatial Cross Attention and Radar Occupancy Information Guided Spatial Cross Attention, Radar Context Aware Spatial Cross Attention, and a task-specific head (Detection / Segmentation Head). The depth distribution-induced spatial cross-attention module and the radar occupancy-induced spatial cross-attention module can accurately transform viewpoint image context features into the BEV space using depth information (depth distribution and radar occupancy information). Subsequently, the subsequent radar context-aware spatial cross-attention module can integrate radar context information into the features. Finally, the head can generate a final result by performing downstream operations, such as detection and / or segmentation, on the BEV features.

[0055] A. Basics

[0056] BEVFormer (Bird's-Eye View Transformer) is a camera-based 3D object detection model that uses backprojection-based viewpoint transformation. For viewpoint transformation, BEV query Points distributed along the z-axis is image context features at time N using camera parameter P It is projected onto. Here, C, X, Y, H, and W represent the channel, BEV grid size, and image resolution, respectively. Features to be sampled based on the projected points are learned through deformable attention. Then, the BEV query, through a set of weighted sampled features, as shown in Equation 1 below It is updated to.

[0057]

[0058] In BEVFormer, due to camera geometry, 3D points of a BEV query are projected along the same ray onto the image feature map at the same location, resulting in the disadvantage of obtaining the same features without distinguishing depth. To address this limitation, DFA3D instead uses a depth distribution and image context features The elevated result of the outer product between them It uses as a value. This allows queries to be projected to different points along the same ray, enabling the retrieval of features with distinct depth values.

[0059] B. Image and Radar Point Cloud Processing

[0060] Camera streams can be processed through the BEVFormer-DFA3D method. For example, six images covering the 360-degree Field of View (FoV) around the vehicle can be used as input, and multiscale image feature maps can be extracted using an image backbone (e.g., ResNet (Residual Network), V-99) and a Feature Pyramid Network (FPN). These image feature maps are then passed through DepthNet to follow the Bird's-Eye View Depth (BEVDepth) and Lift, Splat, Shoot (LSS) methodologies, resulting in a discrete depth distribution. You can obtain . In addition, through the use of multiple CNN (Convolutional Neural Network) layers and a softmax function, occupancy information included and including contextual information You can obtain.

[0061] On the other hand, for radar streams, a point cloud with coordinates in the radar coordinate system is projected onto a viewpoint image to locate corresponding pixels, and can be voxelized into a camera frustum view following the CRN. During this projection process, depth in the camera coordinate system is preserved, allowing for the generation of cylindrical frustum view coordinates (u,v=1,d). In this frustum view, u and v can represent the width and height in the pixel coordinate system, respectively, while d can represent the depth in the camera coordinate system. Subsequently, it passes through PointNet and SECOND (Sparsely Embedded Convolutional Detection) to obtain occupancy information. and radar context information (e.g., Radar Cross Section (RCS) and Doppler velocity) It can extract.

[0062] C. View Transformation

[0063] 1. Radar Occupancy Information-Based Spatial Cross Attention: DFA3D adopts depth-weighted bilinear interpolation operations to solve the same feature problem in the same ray. This operation uses the predicted depth distribution in the image It is based on probabilistic information within. However, there is a problem in that it is difficult to accurately distinguish image context features due to the inherently inaccurate depth information obtained from monocular depth estimation. Therefore, in this embodiment, depth information of image context features can be better distinguished by using radar occupancy information that is sparse but accurate, similar to CRN.

[0064] FIG. 4 illustrates an example of radar occupancy information-based spatial cross-attention according to an embodiment of the present invention. The embodiment of FIG. 4 shows that extended features can be obtained by using the cross product of radar occupancy information and image occupancy information, and the cross product of image context features. Features can be aggregated into a BEV query through a spatial cross-attention operation between the BEV query and the extended features. In FIG. 4, since points (A) and (B) lie on the same ray but are projected at different depths, features distinguishing depth can be encoded.

[0065] More specifically, in the embodiment of FIG. 4, perspective image occupancy information and frustum view radar occupancy information 3D occupancy information by taking the cross product of You can obtain. Next, image context Extended feature map by taking the outer product of You can obtain. Now, the expanded feature map and BEV query Encoded depth-aware image features through the depth-aware spatial cross-attention operation (DFA3D). ...can be obtained. The entire process can be represented as shown in the following mathematical equation 2.

[0066]

[0067] B R In addition, this embodiment utilizes depth distribution to include BEV features encoded with image context ...can be obtained. In this embodiment, these two BEV features can be combined into one BEV feature B IR It can be combined. The combination process can be represented as shown in the following mathematical equation 3.

[0068]

[0069] Here can represent an operation that masks projected points outside the feature map, ψ can represent a flattening operation, ";" can represent a connection in the channel dimension, and ξ can represent a Multilayer Perceptron (MLP) network.

[0070] 2. Radar Context-Aware Spatial Cross-Attention: The previously introduced ROSCA module can use radar to provide occupancy information in the frustum view, which enables better integration of image context features into BEV features. In addition to occupancy information, the radar point cloud provides RCS and Doppler velocity V x ,V y It can provide such radar context information. In order to utilize this radar context information, the present embodiment may perform a spatial cross-attention operation between a radar context feature map and a BEV feature in which an image context was previously encoded. In this case, a 3D point in the BEV feature can be projected onto a frustum-shaped radar context feature with a height of 1.

[0071] As can be observed in Fig. 3(b), different points on the same ray can retrieve different features. This is because radar features are extracted along the (u,d) direction. Furthermore, by using radar context features in the frustum view format instead of radar context features in the BEV format, CRAB can learn consistently without view differences during spatial cross-attention operations, thereby effectively finding correspondences with image features. Table 1 below shows that the design selection according to this embodiment for radar context features in the frustum view format is effective.

[0072]

[0073] In addition, the above-described process can be expressed as shown in mathematical equation 4 below.

[0074]

[0075] Here It can represent a matrix that projects 3D points from a BEV onto a frustum view radar context feature map.

[0076] FIG. 5 illustrates a comparative example of encoded BEV features in an embodiment of the present invention. During view transformation, (a) does not consider depth, so features appear somewhat scattered and unclear in the triangular region (ray region). On the other hand, BEV features (b) of the architecture according to the present embodiment show a clear distinction regarding the depth of the features and demonstrate improved clarity. Thus, after passing through the two modules presented earlier (ROSCA and RCSCA), as can be seen in FIG. 5 (b), the present embodiment can finally obtain BEV features encoded with both image and radar context features. These BEV features can pass through decoders tailored to each downstream task, such as 3D object detection and / or segmentation.

[0077] D. Loss Function

[0078] Detection loss L to train CRAB detection , depth loss L depth and loss of possession L occupancy Three types of losses can be utilized. These three losses can be expressed as shown in Equation 5 below.

[0079]

[0080] Detection Loss L detection It can include L1 loss for bounding box regression and focal loss for classification. Depth loss L depth Binary cross-entropy loss can be used to supervise the depth distribution. Finally, image occupancy loss L occupancy It can be trained using Gaussian focal loss based on a heatmap obtained from a 2D bounding box. This heatmap can be obtained through 3D bounding box annotations.

[0081] 4. Experiment

[0082] A. Datasets and Metrics

[0083] The proposed architecture can be evaluated based on the nuScenes public dataset. For 3D object detection, 6 TP (6 True Positives) and NDS (NuScenes Detection Score) metrics can be used, and mIoU (mean Intersection over Union) can be used for segmentation.

[0084] B. Implementation Details

[0085] BEVFormer-DFA3D can be adopted as the baseline. The temporal self-attention module and detection head used in BEVFormer are used without modification. ResNet101-DCN and VovNet are used as the backbone for imaging. Radar point clouds from six previous sweeps are accumulated along the GRIF Net. Six encoder layers are used, and the BEV grid size is 200×200 as shown in Tables 2, 3, and 4.

[0086]

[0087]

[0088]

[0089] Table 1 shows examples of quantitative results for 3D object detection on the nuScenes validation set. In Table 1, 'L', 'C', and 'R' represent LiDAR, Camera, and Radar, respectively. * indicates results obtained from MMDETECTION3D, and is initialized on the FCOS3D backbone, Each indicates that it was trained with CBGS, and the bolded and underlined values ​​represent the best and second-best results between C and C&R. Additionally, VT stands for View Transformation.

[0090] Table 2 shows an example of the results of 3D object detection on the nuScenes test set. Bold and underlined values ​​represent the highest and second-best values.

[0091] Table 3 shows examples of analysis results for various lighting and weather conditions. Values ​​in bold and underlined indicate the highest and second-best values.

[0092] Additionally, Tables 1, 5, and 6 use three encoder layers and a 150×150 BEV grid. To ensure a fair comparison with the baseline, data augmentation, future frames, CBGS, and TTA (Test Time Augmentation) are not used. The experiment is conducted for 24 epochs, using an AdamW optimizer and a 2×10 -4 Train on 4 NVIDIA RTX3090 GPUs using a learning rate.

[0093]

[0094]

[0095] Table 5 shows an example of the segmentation results for the NUScenes dataset. Bold and underlined values ​​represent the highest and second-best values.

[0096] Table 6 shows examples of the results of removal experiments for ROSCA and RCSCA.

[0097] C. Evaluation Results

[0098] 1. 3D Object Detection: Tables 2 and 3 show the experimental results performed on nuScenes' validation and test sets, respectively. CRAB achieved significant improvements over camera-only methods. Furthermore, CRAB outperforms baselines using camera-only and camera-radar fusion methods. This suggests that CRAB effectively fuses cameras and radar based on back-projection compared to other methods. Additionally, CRAB demonstrates improved performance compared to other fusion methods, highlighting the potential for radar fusion in back-projection-based approaches. Finally, compared to radar-only methods, it shows slightly lower or superior performance, suggesting the potential for camera and radar fusion.

[0099] Figure 6 illustrates an example of the qualitative results of 3D object detection in an embodiment of the present invention. When observing the triangular region (ray region), BEVFormer-DFA3D and TransCAR predict multiple instances for a single object, resulting in many false positives. In contrast, it can be seen that CRAB significantly reduces false positives. More specifically, Figure 6 shows a qualitative comparison between the baseline, TransCAR, and CRAB. Looking at the triangular region centered on the ray region, it can be seen that the baseline method predicts multiple instances for a single object (the real object) despite using a depth distribution. This phenomenon also occurs in TransCAR, suggesting that although it uses the same view transformation (backprojection), it does not account for depth ambiguity. On the other hand, CRAB effectively mitigates depth ambiguity by utilizing accurate depth information from the radar, demonstrating a noticeable reduction in false positives.

[0100] FIG. 7 is a diagram illustrating an example of the qualitative results of BEV splitting in the nuScenes verification set in one embodiment of the present invention.

[0101] 2. Segmentation: Segmentation experiments were conducted as shown in Table 5 and Figure 7. The segmentation head consists of multiple CNN layers and predicts three classes: separator, pedestrian, and boundary. Cross-entropy loss is used for segmentation training. Compared to BEVFormer, CRAB showed improved performance across all classes and exhibited similar results when compared to other state-of-the-art models. This indicates that CRAB can effectively perform various recognition tasks by generating general and sophisticated BEV features.

[0102] D. Analysis and Loss Research

[0103] 1. Component Analysis: A loss study was conducted to evaluate the effectiveness of the modules (ROSCA and RCSCA). As shown in Table 6, (A) represents the baseline BEVFormer-S-DFA3D. Comparing (A) and (B), it can be seen that using only the depth distribution obtained from images makes it difficult to solve the same-ray same-feature problem, and that utilizing radar occupancy allows for obtaining more accurate depth information. Comparing (B) and (D), performance is improved by integrating radar context information through spatial attention. It can be confirmed that radar context (RCS, Doppler velocity) information helps in understanding the 3D scene. Finally, it is speculated that the reason the improvement from (B) to (D) is greater than the improvement from (C) to (D) is that RCSCA has already learned the sampling offset to refine image features with depth ambiguity obtained previously.

[0104] 2. Analysis of Radar Contextual Feature Views: Table 1 presents the analysis of RCSCA. RCSCA was not used in this experiment. Comparing (a) and (c), it can be seen that the performance of (c) is significantly superior. This supports the idea that radar contextual features should be extracted from the frustum view due to the inaccuracy of radar elevation information. Additionally, the performance of (a) is considerably lower than the baseline ((A) Table 6), suggesting that appropriate fusion is required. (b) shows better performance than (c). This is a result of CRAB extracting radar features from the same perspective as the image, eliminating view mismatch, and making it easier for the network to learn.

[0105] 3. Lighting and Weather Conditions: As shown in Table 4, NDS was measured for the validation set under various weather and lighting conditions. Thanks to the radar's robustness against adverse weather, CRAB consistently outperforms the camera-only baseline in all scenarios. Compared to other camera-radar fusion models, CRAB demonstrates superior performance. This demonstrates the effectiveness of a fusion method that leverages the strengths of various modalities. Furthermore, a comparison of CenterPoint-P and CRAB under rain and night conditions confirms that the radar's robustness stands out more in challenging environments.

[0106] FIG. 8 is a diagram illustrating an example of a three-dimensional object detection method according to an embodiment of the present invention. The three-dimensional object detection method according to the present embodiment may be performed by a CRAB-based three-dimensional object detection system, which is a camera-radar fusion-based three-dimensional object detection (detection and / or segmentation) model that mitigates depth ambiguity by utilizing the back-projection described above. Such an object detection system may be implemented by at least one computer device. For example, at least one computer device may be implemented such that at least one processor controls at least one computer device by executing a control instruction according to the code of an operating system or the code of at least one computer program contained in at least one memory. As a more specific example, at least one processor may control at least one computer device according to a control instruction provided by the code stored in at least one computer device so that the three-dimensional object detection system performs steps (810 to 860) included in the method of FIG. 8.

[0107] In step (810), the 3D object detection system can collect images from a camera. From the images, image feature maps containing contextual information, tensors containing depth distribution information, and tensors containing occupancy information in perspective can be extracted through multiple CNNs. The image backbone for such extraction has been described in detail.

[0108] In step (820), the 3D object detection system can collect radar points from the radar. From the cloud of radar points, a feature map containing contextual information of the radar and a tensor containing occupancy information can be extracted. At this time, in order to extract contextual information and occupancy information in the width of the image and the depth of the camera, the cloud of radar points can be projected onto an image coordinate system as described in the subsequent step (830).

[0109] In step (830), the 3D object detection system can project a cloud of collected radar points onto an image coordinate system for the image in order to correspond the pixels of the collected image with the collected radar points. As previously described, as the cloud of radar points is projected onto the image coordinate system, contextual information and occupancy information can be extracted in the width of the image and the depth of the camera.

[0110] In step (840), the 3D object detection system can generate BEV features by converting the contextual features of the image into BEV space. For example, the 3D object detection system can generate a BEV feature map using a back-projection-based view transformation on the image feature map of the image, and at this time, the contextual features of the image can be integrated into the BEV feature map. As a more specific example, the 3D object detection system can calculate the cross product of radar occupancy information extracted from the radar point cloud and image occupancy information extracted from the image, and can calculate an extended feature map using the calculated cross product and the contextual features of the image. Subsequently, the 3D object detection system can generate a first BEV feature encoded with depth-aware image features through a depth-aware space cross-attention operation between the extended feature map and the BEV query. Here, the radar occupancy information may include occupancy information in a frustum view generated by voxelizing the radar point cloud into a camera frustum view according to the CRN. Additionally, the depth-aware spatial cross-attention operation may include a deformable attention operation using an extended feature map as the key, camera parameters as the value, and a BEV query as the query. Furthermore, the 3D object detection system may generate a second BEV feature encoded with the contextual features of the image using the depth distribution predicted in the image. For example, the 3D object detection system may generate the second BEV feature through a depth-aware spatial cross-attention operation based on the depth distribution and image distance information of the image. In this case, the 3D object detection system may generate a BEV feature by combining the first BEV feature and the second BEV feature.

[0111] In step (850), the 3D object detection system can integrate contextual information extracted from the radar point cloud into BEV features. For example, the 3D object detection system can integrate the radar contextual information into the BEV feature map through a context-aware spatial cross-attention operation between the radar contextual information and the BEV features. Here, the context-aware spatial cross-attention operation may include a deformable attention operation using the radar contextual information as key and value and the BEV features as query. At this time, the contextual information as key may include RCS and Doppler velocity, and the contextual information as value may include a matrix that projects the radar points from the BEV to the contextual feature map in the frustum view. At this time, since the radar contextual information is extracted in the width direction of the image coordinate system and the depth direction of the camera coordinate system, even if 3D points on the same ray are projected, they reach different points, so a feature that distinguishes depth can be integrated into the BEV features.

[0112] In step (860), the 3D object detection system can detect or segment 3D objects using BEV features. A task-specific head for detecting or segmenting 3D objects using BEV features has been described previously.

[0113] At least steps (830), (840), and (850) of the previously described steps (810 to 860) can be performed using CRAB, the previously described 3D object detection model. As previously described, CRAB can be trained using detection loss, depth loss, and occupancy loss. Here, detection loss may include L1 loss for bounding box regression and focal loss for classification, depth loss may include binary cross-entropy loss to supervise depth distribution, and occupancy loss may include Gaussian focal loss based on a heatmap obtained from 2D bounding boxes.

[0114] Thus, according to embodiments of the present invention, a new camera-radar fusion-based 3D object detection method and system using backprojection that alleviates depth ambiguity can be provided.

[0115] FIG. 9 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, a three-dimensional object detection system using CRAB described above may be implemented by at least one computer device, wherein each of the at least one computer device may correspond to the computer device (900) of FIG. 9. As illustrated in FIG. 9, the computer device (900) may include memory (910), a processor (920), a communication interface (930), and an input / output interface (940). The memory (910) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, the non-perishable mass storage device such as ROM and the disk drive may be included in the computer device (900) as a separate permanent storage device distinct from the memory (910). Additionally, an operating system and at least one program code may be stored in the memory (910). These software components may be loaded into the memory (910) from a computer-readable recording medium separate from the memory (910). This separate computer-readable recording medium may include computer-readable recording media such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, the software components may be loaded into the memory (910) via a communication interface (930) rather than a computer-readable recording medium. For example, the software components may be loaded into the memory (910) of the computer device (900) based on a computer program installed by files received through a network (Network, 960).

[0116] The processor (920) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (920) via memory (910) or a communication interface (930). For example, the processor (920) may be configured to execute instructions received according to program code stored in a recording device such as memory (910).

[0117] The communication interface (930) may provide a function for the computer device (900) to communicate with other devices through a network (960). For example, requests, commands, data, files, etc. generated by the processor (920) of the computer device (900) according to program code stored in a recording device such as memory (910) may be transmitted to other devices through the network (960) under the control of the communication interface (930). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (900) through the communication interface (930) of the computer device (900) via the network (960). Signals, commands, data, etc. received through the communication interface (930) may be transmitted to the processor (920) or memory (910), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (900) may further include.

[0118] The input / output interface (940) may be a means for interfacing with an input / output device (I / O device, 950). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (940) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device (950) may be composed of a computer device (900) and a single device.

[0119] Additionally, in other embodiments, the computer device (900) may include fewer or more components than the components of FIG. 9. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (900) may be implemented to include at least some of the input / output devices (950) described above, or may include other components such as a transceiver, a database, etc.

[0120] The system or device described above may be implemented as a hardware component, or a combination of a hardware component and a software component. For example, the device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0121] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0122] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware components, and is not limited to a medium directly connected to a computer system, but may also exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0123] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0124] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A method for detecting three-dimensional objects in a three-dimensional object detection system implemented with at least one computer device, The above at least one computer device includes at least one processor, and A step of collecting an image from a camera by the above at least one processor; A step of collecting radar points from a radar by the above-mentioned at least one processor; A step of projecting a cloud of the collected radar points onto an image coordinate system for the image by the at least one processor to correspond the pixels of the collected image with the collected radar points to each other; A step of generating BEV features by converting the context features of the image into a BEV (Bird's-Eye View) space by the at least one processor; A step of integrating context information extracted from the radar point cloud into the BEV feature by the at least one processor; and A step of detecting or segmenting a 3D object using the BEV feature by the above at least one processor A three-dimensional object detection method including 2. In Paragraph 1, The step of generating the above BEV features is, A 3D object detection method characterized by generating a BEV feature map using a back-projection-based view transformation for an image feature map of the above image.

3. In Paragraph 2, The step of generating the above BEV features is, A 3D object detection method characterized by generating a BEV feature map by integrating contextual features of the above image into the BEV feature map.

4. In Paragraph 3, The step of generating the above BEV features is, A step of calculating the outer product of radar occupancy information extracted from the cloud of the radar point and image occupancy information extracted from the image; A step of calculating an extended feature map using the above cross product and the contextual features of the above image; and A step of generating a first BEV feature encoded with depth-aware image features through a depth-aware spatial cross attention operation between the extended feature map and the BEV query. A three-dimensional object detection method characterized by including 5. In Paragraph 4, The above radar occupancy information includes occupancy information in a frustum view generated by voxelizing the cloud of the above radar point into a camera frustum view according to the CRN, and A 3D object detection method characterized in that the depth perception spatial cross-attention operation includes a deformable attention operation using the extended feature map as a key, the camera parameters as a value, and the BEV query as a query.

6. In Paragraph 3, The step of generating the above BEV features is, A step of generating a second BEV feature encoded with contextual features of the image using the depth distribution predicted in the image above. A three-dimensional object detection method characterized by further including 7. In Paragraph 6, The step of generating the above second BEV feature is, A 3D object detection method characterized by generating the second BEV feature through a depth-aware spatial cross attention operation based on the depth distribution and image distance information of the image.

8. In Paragraph 2, The step of integrating the extracted contextual information into the BEV features is, A 3D object detection method characterized by integrating the context information of the radar into the BEV feature map through a context-aware spatial cross attention operation between the context information of the radar and the BEV features.

9. In Paragraph 8, A 3D object detection method characterized in that the above context-aware spatial cross-attention operation includes a deformable attention operation using the context information of the radar as key and value and the BEV feature as a query.

10. In Paragraph 1, The steps of projecting the cloud of the collected radar points onto an image coordinate system for the image, generating the BEV features, and integrating the extracted context information into the BEV features are: A 3D object detection method characterized by being performed using a 3D object detection model trained using detection loss, depth loss, and occupancy loss.

11. In Paragraph 10, The above detection loss includes L1 loss for bounding box regression and focal loss for classification, and The above depth loss includes binary cross-entropy loss to supervise the depth distribution, The above occupation loss includes a Gaussian focal loss based on a heatmap obtained from a 2D bounding box. A three-dimensional object detection method characterized by 12. A computer-readable recording medium having a computer program recorded thereon for executing the method of any one of paragraphs 1 through 11 on a computer device.

13. A computer program stored on a computer-readable recording medium for executing a three-dimensional object detection method on a computer device combined with a computer device implementing a three-dimensional object detection system, The above 3D object detection method is, Step of collecting images from a camera; Step of collecting radar points from the radar; A step of projecting a cloud of the collected radar points onto an image coordinate system for the image in order to correspond the pixels of the collected image with the collected radar points; A step of generating BEV features by converting the contextual features of the above image into a BEV (Bird's-Eye View) space; A step of integrating context information extracted from the radar point cloud into the BEV features; and A step of detecting or segmenting a 3D object using the above BEV features A computer program characterized by including 14. In Paragraph 13, The step of generating the above BEV features is, A computer program characterized by generating a BEV feature map using a back-projection-based view transformation for an image feature map of the above image, wherein the contextual features of the above image are integrated into the BEV feature map to generate the BEV feature map.

15. In Paragraph 14, The step of generating the above BEV features is, A step of calculating the outer product of radar occupancy information extracted from the cloud of the radar point and image occupancy information extracted from the image; A step of calculating an extended feature map using the above cross product and the contextual features of the above image; and A step of generating a first BEV feature encoded with depth-aware image features through a depth-aware spatial cross attention operation between the extended feature map and the BEV query. A computer program characterized by including 16. In Paragraph 14, The step of generating the above BEV features is, A step of generating a second BEV feature encoded with contextual features of the image using the depth distribution predicted in the image above. Includes, The step of generating the above second BEV feature is, Generating the second BEV feature through a depth-aware spatial cross attention operation based on the depth distribution and image distance information of the image. A computer program characterized by 17. In Paragraph 15, The step of integrating the above context information into the above BEV features is, integrating the context information of the radar into the BEV feature through a context-aware spatial cross attention operation between the context information of the radar and the BEV feature, and The above context-aware spatial cross-attention operation includes a deformable attention operation using the context information of the radar as key and value and the BEV feature as a query. A computer program characterized by 18. In a three-dimensional object detection system implemented with at least one computer device, The above at least one computer device includes at least one processor, and By the above at least one processor, Collect images from the camera, Collect radar points from the radar, In order to correspond the pixels of the collected image with the collected radar points, the cloud of the collected radar points is projected onto an image coordinate system for the image, and The contextual features of the above image are converted into a BEV (Bird's-Eye View) space to generate BEV features, and Contextual information extracted from the above radar point cloud is integrated into the above BEV features, and Detecting or segmenting 3D objects using the above BEV features A 3D object detection system characterized by 19. In Paragraph 18, To generate the above BEV features, by the at least one processor, Generating a BEV feature map using a back-projection-based view transformation for an image feature map of the above image, wherein the contextual features of the above image are integrated into the BEV feature map to generate the BEV feature map. A 3D object detection system characterized by 20. In Paragraph 19, To generate the above BEV features, by the at least one processor, Calculate the outer product of the radar occupancy information extracted from the cloud of the above radar points and the image occupancy information extracted from the above image, and Calculate an extended feature map using the above cross product and the contextual features of the above image, and Generating a first BEV feature encoded with depth-aware image features through depth-aware spatial cross attention operations between the extended feature map and the BEV query. A 3D object detection system characterized by

Images