Multimodal 3D Object Detection Using Temporal Structure Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for depth estimation in autonomous driving systems face challenges such as high computational demand, difficulty in mapping to hardware implementations, and inaccurate confidence estimation of object detection features.
Innovation Solution
The proposed solution involves a Structure from Motion (SfM) based approach that uses voxel representations from images and LIDAR point clouds to determine object positions in a bird's eye view (BEV) representation. This method extracts features from images and point clouds, establishes correspondences between voxels over time, and uses pose data for triangulation to generate an accurate BEV representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If test-time refinement techniques are used to update network parameters for depth estimation, then measurement precision is improved, but productivity deteriorates due to costly multiple forward and backward passes
Solution Approach 1:
The patent pre-computes and stores depth maps during the training phase using ground truth annotations. These pre-computed depth maps are then reused during test-time for refinement without requiring costly forward-backward passes, thus maintaining high precision while improving computational efficiency during deployment
Solution Approach 2:
The patent creates a copy of the depth information from ground truth data during training and stores it as reference depth maps. During test-time, these copied depth maps are used directly for refinement purposes, avoiding the need to re-compute them through expensive optimization procedures
2Measurement precision
If recurrent neural networks process multiple frames during training, then measurement precision is improved, but productivity deteriorates due to computational demand of extracting features from multiple frames
Solution Approach 1:
The patent extracts only the essential depth information from multiple frames during training and stores it as pre-computed depth maps. During test-time, only these extracted depth maps are used for refinement rather than processing entire frames again, significantly reducing computational demand while maintaining precision
Solution Approach 2:
The patent performs the computationally intensive feature extraction from multiple frames during the training phase and stores the results. During test-time, the pre-extracted features and depth maps are reused, eliminating the need to repeat the expensive feature extraction process
3Productivity
If end-to-end cost volume approaches are used to aggregate information, then productivity is improved compared to other methods, but device complexity increases due to difficulty in mapping to hardware implementations
Solution Approach 1:
The patent segments the depth estimation process into distinct modules: feature extraction, cost volume computation, and depth refinement. Each module is designed with clear interfaces and can be independently optimized for hardware implementation, reducing overall system complexity while maintaining computational efficiency
Solution Approach 2:
The patent transforms the 4D cost volume computation into a series of 3D operations by fixing one dimension during hardware implementation. This dimensional transformation allows the system to maintain the computational efficiency of end-to-end approaches while making the operations more suitable for standard hardware architectures
Data Source
AI summary
An example device for detecting objects through processing of media data, such as image data and point cloud data, includes a processing system configured to form voxel representations of a real-world three-dimensional (3D) space using images and point clouds captured for the 3D space at consecutive time steps, extract image and/or point cloud features for voxels in voxel representations of the 3D space, determine correspondences between the voxels at consecutive time steps according to similarities between the extracted features, and determine positions of objects in the 3D space using the correspondences between the voxels. For example, the processing system may perform triangulation according to positions of a moving object to positions of the voxels at the time steps. In this manner, the processing system may generate an accurate bird's eye view (BEV) representation of the real-world 3D space.