DuoSpaceNet 3D Detection via BEV and PV Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image processing methods for autonomous vehicles struggle to effectively combine 2D and 3D feature representations from bird's-eye-view (BEV) and perspective-view (PV) spaces, leading to suboptimal 3D detection and segmentation performance.
Innovation Solution
The proposed DuoSpaceNet technique fuses BEV and PV features through a duo-space decoder, using partial cross-attention layers and feature divergence enhancement to create a unified representation for 3D object detection and map segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If only 2D perspective view features are used for object detection, then the computational complexity is lower, but the 3D detection precision and segmentation performance deteriorate
Solution Approach 1:
The patent merges 2D perspective view features and 3D bird's eye view features into a unified feature representation. The encoder processes both 2D image features and 3D BEV features through shared weight matrices, allowing the system to leverage the computational efficiency of 2D features while incorporating the spatial accuracy of 3D features for improved detection precision.
Solution Approach 2:
The patent transforms 2D perspective view features into 3D bird's eye view features through a dimensionality transformation process. The 2D features are projected and aggregated into 3D space using camera intrinsics and extrinsics, enabling the system to benefit from both 2D computational efficiency and 3D spatial understanding.
2Measurement precision
If only 3D bird's eye view features are used for object detection, then the 3D detection precision is improved, but the computational complexity increases
Solution Approach 1:
The patent performs preliminary 2D feature extraction and processing before transforming them into 3D BEV features. By pre-processing 2D features with lightweight convolutions and then aggregating them into 3D space, the system reduces the computational burden of direct 3D feature extraction while maintaining detection precision.
Solution Approach 2:
The patent uses 2D perspective view features as a computational proxy or copy that can be transformed into 3D representations. Instead of directly extracting complex 3D features from raw images, the system creates 2D feature copies that are then projected into 3D space, reducing overall computational complexity.
3Reliability
If 2D and 3D feature representations are combined, then the detection performance is improved, but the feature fusion complexity increases
Solution Approach 1:
The patent employs a universal encoder with shared weight matrices that processes both 2D and 3D features through the same transformation layers. This multi-functional architecture allows the same computational blocks to handle different feature types, simplifying the fusion process while improving detection reliability through complementary feature integration.
Solution Approach 2:
The patent changes the parameter representation by using camera intrinsics and extrinsics to transform 2D feature coordinates into 3D BEV space. By parameterizing the transformation using camera calibration data, the system achieves seamless feature fusion without complex alignment algorithms, improving detection performance while controlling fusion complexity.
4Speed
If sparse feature map transformation to 3D space is performed, then the processing speed is faster, but the segmentation accuracy deteriorates
Solution Approach 1:
The patent applies iterative refinement modules that continuously improve the dense feature map in 3D space. The refinement process operates in multiple passes, each pass enhancing the segmentation accuracy by refining feature representations while maintaining the initial processing speed advantage of sparse-to-dense transformation.
Solution Approach 2:
The patent incorporates feedback mechanisms where the dense feature map is repeatedly refined using the same transformation and refinement pipeline. The output of each refinement pass feeds back into the next pass, allowing the system to achieve high segmentation accuracy through iterative improvement while maintaining efficient processing through the reusable transformation pipeline.
Data Source
AI summary
An image processing method includes performing, using images obtained from one or more sensors onboard a vehicle, a 2-dimensional (2D) feature extraction; performing, a 3-dimensional (3D) feature extraction on the images; detecting objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction.