Single-Frame 4D Sensor Fusion for Vehicle Object and Velocity Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current vehicle navigation systems face challenges in accurately detecting objects and estimating their velocities using conventional RADAR sensors, which provide coarse resolutions and lack elevation angle information, and existing 3D object detection methods struggle due to lower resolutions of 3D input data and high complexity in estimating oriented bounding boxes.
Innovation Solution
The integration of camera, imaging RADAR, and LiDAR data fusion using deep neural networks (DNNs) for four-dimensional (4D) object detection, where a single frame of each data type is synchronized to provide object classification, 3D position, dimensions, direction, and velocity, leveraging the strengths of each sensor type.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional RADAR sensors are used for object detection, then the system can detect objects, but the resolution is coarse and elevation angle information is missing
Solution Approach 1:
The patent combines conventional RADAR, LiDAR, and camera sensors into a unified detection system. The conventional RADAR provides velocity information through Doppler effects, LiDAR provides high-resolution 3D spatial data through laser time-of-flight measurement, and cameras provide visual classification. This merging resolves the contradiction by achieving high measurement precision through multi-sensor fusion while distributing the complexity across specialized components rather than requiring a single complex sensor.
Solution Approach 2:
The integrated system performs multiple functions simultaneously: object detection, velocity estimation, 3D localization, and classification. The conventional RADAR contributes velocity measurement capability, LiDAR contributes precise spatial positioning, and cameras contribute visual identification. This multi-functionality approach resolves the contradiction by making each sensor contribute its strength to the overall system performance.
2Measurement precision
If LiDAR point cloud data is used for 3D object detection, then position and dimension information can be obtained, but the input data resolution is lower and the complexity of estimating oriented bounding boxes increases
Solution Approach 1:
The patent introduces an intermediate processing stage where LiDAR point cloud data is converted into oriented bounding box representations before final object detection. This intermediary representation simplifies the detection task by providing pre-processed 3D boundary information, reducing the algorithmic complexity while maintaining measurement precision. The oriented bounding boxes serve as a bridge between raw point cloud data and final object classification.
Solution Approach 2:
The detection process is segmented into distinct stages: LiDAR point cloud processing, oriented bounding box estimation, and final object detection. By dividing the complex task into manageable segments, the system achieves accurate 3D position measurement without being overwhelmed by the full complexity of direct 3D object detection from raw point clouds.
3Loss of information
If multiple sensor frames are processed separately, then each sensor provides its data, but velocity estimation requires additional tracking algorithms across multiple frames
Solution Approach 1:
The patent merges velocity information extraction into the single-frame detection process by combining RADAR Doppler velocity measurements with object detection results. Instead of requiring separate tracking algorithms across multiple frames, the system integrates velocity estimation directly into the detection pipeline through sensor fusion, reducing algorithmic complexity while maintaining complete velocity information.
Solution Approach 2:
The system performs preliminary velocity measurement using RADAR Doppler effects before final object detection. This preliminary action provides velocity information in advance, eliminating the need for complex post-detection tracking algorithms to estimate velocity from multiple frames.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enhances object detection accuracy and velocity estimation by combining high-resolution imaging RADAR data, precise LiDAR information, and color perception from cameras, overcoming limitations of conventional systems and improving performance in adverse weather conditions.
Implementation Method 1
Vehicles use sensors to provide information about the surroundings of the vehicle such as detecting objects. This information is used to augment or automate vehicle operations. Sensors include, but are not limited to, RADAR sensors, camera sensors, LiDAR sensors, etc.
Implementation Method 2
each frame of imaging RADAR detection includes doppler information for the object from which the velocity of the object can be determined
Implementation Method 3
LiDAR systems provide a LiDAR point cloud of reflective points, wherein the LiDAR point cloud is much denser than the RADAR point cloud and provides a better resolution
Data Source
AI summary
Embodiments of the present disclosure are directed to a method for object detection. The method includes receiving sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem. The sensor data is received simultaneously and within one frame for each of the subsystems. The method also includes extracting one or more feature representations of the objects from camera image data, LiDAR point cloud data and imaging RADAR point cloud data and generating image feature maps, LiDAR feature maps and imaging RADAR feature maps. The method further includes combining the image feature maps, the LiDAR feature maps and the imaging RADAR feature maps to generate merged feature maps and generating object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.