Single-Frame 4D Object Detection With Camera, LiDAR, and Imaging RADAR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current vehicle navigation systems face challenges in detecting objects and estimating their velocities using traditional 2D and 3D object detection methods, particularly due to the low resolution of RADAR sensors and the complexity of estimating oriented bounding boxes, as well as the limitations of LiDAR and camera systems in providing accurate depth information.
Innovation Solution
A four-dimensional (4D) object detection system that fuses camera, imaging RADAR, and LiDAR data using deep neural networks (DNNs) to simultaneously process camera image data, imaging RADAR point cloud data, and LiDAR point cloud data, enabling the classification of objects, determination of their 3D position, dimensions, orientation, and velocity within a single frame.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional RADAR sensors are used, then the system can detect objects, but the resolution is very coarse and elevation angle information is not included
Solution Approach 1:
The patent combines conventional RADAR, LiDAR, and camera systems into an integrated sensor suite. The conventional RADAR provides coarse detection coverage, while LiDAR adds high-resolution 3D point cloud data and the camera provides 2D image data. This merging allows the system to achieve improved measurement precision through multi-sensor fusion without requiring a single complex high-resolution RADAR system.
Solution Approach 2:
The patent introduces imaging RADAR as an intermediary sensor that bridges the gap between conventional RADAR and LiDAR. Imaging RADAR provides intermediate resolution with elevation angle information, serving as a mediator that enhances the overall system precision while maintaining reasonable complexity through its specialized function in the sensor hierarchy.
2Measurement precision
If LiDAR point cloud data is used for 3D object detection, then position and dimension information can be obtained, but the input data resolution is lower and estimating oriented bounding box is highly complex
Solution Approach 1:
The patent uses camera 2D detection algorithms as an intermediary approach to simplify 3D object detection. Instead of directly estimating complex oriented bounding boxes from low-resolution LiDAR point clouds, the system first performs 2D object detection on camera images to obtain initial object locations and classifications, then uses these results to guide and simplify the 3D bounding box estimation process from LiDAR data.
Solution Approach 2:
The detection process is segmented into multiple stages: first performing 2D object detection and classification on camera images, then using those results to guide 3D object detection on LiDAR point clouds. This segmentation breaks down the highly complex single-step 3D detection problem into more manageable sequential steps, reducing overall algorithmic complexity.
3Loss of information
If a tracking algorithm is used to derive object velocities, then velocity information can be obtained, but multiple frames are required which increases time consumption
Solution Approach 1:
The patent introduces imaging RADAR doppler information as an intermediary source for velocity estimation. Instead of relying solely on multi-frame tracking algorithms that require temporal processing, the system uses the doppler shift measurements from imaging RADAR to directly estimate object velocities from single-frame data, significantly reducing processing time while maintaining velocity information completeness.
4Measurement precision
If 2D object detection algorithms are used, then remarkable progress has been achieved, but they have not been transferred well for 3D object detection
Solution Approach 1:
The patent leverages the successful 2D object detection algorithms by applying them to camera images, then using those 2D detection results as a foundation for 3D object detection. The system transitions from 2D to 3D by projecting 2D detected objects into 3D space and refining their parameters using LiDAR point cloud data, effectively adapting 2D algorithm strengths to the 3D detection domain through dimensional transformation.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This system enhances object detection capabilities by providing accurate 4D information (position and velocity) of objects, improving detection accuracy and reliability in various weather conditions, and enabling more effective autonomous vehicle operations.
Implementation Method 1
each frame of imaging RADAR detection includes doppler information for the object from which the velocity of the object can be determined
Implementation Method 2
RADAR systems provide a RADAR point cloud of radio-reflective points
Implementation Method 3
LiDAR systems provide a LiDAR point cloud of reflective points, wherein the LiDAR point cloud is much denser than the RADAR point cloud and provides a better resolution
Data Source
AI summary
Embodiments of the present disclosure are directed to a method for object detection. The method includes receiving sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem. The sensor data is received simultaneously and within one frame for each of the subsystems. The method also includes extracting one or more feature representations of the objects from camera image data, LiDAR point cloud data and imaging RADAR point cloud data and generating image feature maps, LiDAR feature maps and imaging RADAR feature maps. The method further includes combining the image feature maps, the LiDAR feature maps and the imaging RADAR feature maps to generate merged feature maps and generating object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.