Multi-Sensor Fusion for 360-Degree Object Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep learning-based object recognition techniques using single sensors are prone to inaccuracies due to sensor errors or insufficiencies, and effective fusion of multi-sensor data is challenging due to different coordinate systems.
Innovation Solution
A method and apparatus that fuse multi-sensor information from cameras, LiDAR, and radar using a deep learning network, converting feature maps from unique sensor coordinate systems to an integrated 3D coordinate system for accurate and robust object detection and recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single sensor information is used for object detection, then device complexity is reduced, but recognition reliability deteriorates due to sensor errors and insufficiencies
Solution Approach 1:
The patent combines data from multiple sensors (LiDAR, radar, camera) into a unified detection system. The sensor fusion module integrates information from these different sources to perform object detection, thereby improving reliability through redundant and complementary measurements while managing the complexity through systematic data integration.
Solution Approach 2:
The patent creates a composite information structure by fusing data from heterogeneous sensor types. The detection result combines features from LiDAR point clouds, radar measurements, and camera images, forming a composite detection output that leverages the strengths of each sensor type to overcome individual limitations.
2Reliability
If multi-sensor data fusion is implemented, then object recognition reliability is improved, but device complexity increases due to different coordinate systems and data integration requirements
Solution Approach 1:
The patent introduces a coordinate transformation module as an intermediary that converts data from various sensor coordinate systems into a unified reference frame. This mediator handles the complexity of coordinate alignment, allowing the fusion module to work with standardized data without directly managing the complexity of multiple coordinate systems.
Solution Approach 2:
The patent divides the data fusion process into distinct functional modules: coordinate transformation, feature extraction, and fusion. Each module handles a specific aspect of the integration process, making the overall complex system manageable through modular design where each segment can be independently optimized and maintained.
3Measurement precision
If deep learning-based feature extraction is used, then measurement precision is improved, but loss of information increases due to data representation transformations
Solution Approach 1:
The patent performs preliminary feature extraction and coordinate transformation on raw sensor data before fusion. By pre-processing each sensor's data to extract relevant features and transform to a common coordinate system, the system prepares information in an optimized format that reduces information loss during the subsequent fusion and detection stages.
Solution Approach 2:
The patent transforms multi-dimensional sensor data (point clouds, images, radar ranges) into a unified feature space through dimensionality transformation. The coordinate transformation module and feature extraction processes convert data from different dimensional representations into a common feature dimension, enabling precise fusion while preserving essential information through appropriate feature selection.
Data Source
AI summary
Presented are a method and a device for multi-sensor data-based fusion information generation for 360-degree detection and recognition of a surrounding object. The present invention proposes a method for multi-sensor data-based fusion information generation for 360-degree detection and recognition of a surrounding object, the method comprising the steps of: acquiring a feature map from a multi-sensor signal by using a deep neural network; converting the acquired feature map into an integrated three-dimensional coordinate system; and generating a fusion feature map for performing recognition by using the converted integrated three-dimensional coordinate system.