Multidimensional Sensor Fusion for Efficient Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches for object detection in autonomous systems, particularly those using 3D data, face challenges with excessive computational costs and latency, making real-time or near real-time deployment infeasible.
Innovation Solution
The system leverages multidimensional sensor data by combining 2D and 3D data for computationally efficient object detection. This is achieved by using deep neural networks that incorporate information from both 2D image data and 3D point cloud data, and by filtering out irrelevant 3D data using regions of interest generated from 2D data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D data is used for object detection, then measurement precision is improved, but computing resources increase excessively
Solution Approach 1:
The patent segments the 3D point cloud data into multiple regions based on 2D image data, processing only relevant regions rather than the entire point cloud. This division allows the system to maintain high detection accuracy in critical areas while reducing overall computational burden by excluding irrelevant regions.
Solution Approach 2:
The patent integrates 2D image data with 3D point cloud data, using the 2D dimension to guide and filter the 3D processing. By projecting 3D points onto 2D image planes and using 2D object detection results to identify relevant regions, the system leverages the complementary strengths of both dimensional spaces to reduce computational complexity while maintaining accuracy.
2Measurement precision
If 3D data is used for object detection, then measurement precision is improved, but processing time increases excessively
Solution Approach 1:
The patent performs preliminary 2D object detection on image data before processing the corresponding 3D point cloud regions. This preliminary action identifies regions of interest that contain potential objects, allowing the system to pre-filter and prioritize which 3D regions require detailed processing, thereby reducing overall processing time while maintaining detection accuracy.
Solution Approach 2:
The patent divides the 3D point cloud into multiple regions corresponding to detected 2D objects, processing each region separately and in parallel when possible. This segmentation enables the system to focus computational resources on relevant regions only, significantly reducing the time required to process entire point clouds while maintaining high detection precision.
3Reliability
If multidimensional data fusion is implemented, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent uses 2D image data as an intermediary to guide 3D point cloud processing. The 2D detection results serve as a mediator that identifies and filters relevant 3D regions, simplifying the data fusion process. This intermediary approach allows the system to leverage multiple data dimensions for improved reliability without requiring complex simultaneous processing of all multidimensional data.
Solution Approach 2:
The patent projects 3D point cloud data onto 2D image planes, creating a unified representation that simplifies fusion. By using 2D projections as an intermediate representation, the system can integrate information from multiple dimensions more easily, improving detection reliability while avoiding the complexity of direct high-dimensional data fusion.
Data Source
AI summary
In various examples, a two-dimensional (2D) and three-dimensional (3D) deep neural network (DNN) is implemented to fuse 2D and 3D object detection results for classifying objects. For example, regions of interest (ROIs) and/or bounding shapes corresponding thereto may be determined using one or more region proposal networks (RPNs)—such as an image-based RPN and/or a depth-based RPN. Each ROI may be extended into a frustum in 3D world-space, and a point cloud may be filtered to include only points from within the frustum. The remaining points may be voxelated to generate a volume in 3D world space, and the volume may be applied to a 3D DNN to generate one or more vectors. The one or more vectors, in addition to one or more additional vectors generated using a 2D DNN processing image data, may be applied to a classifier network to generate a classification for an object.