3D Bounding Box Refinement Using Vision and LiDAR Depth Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous vehicle planning systems face challenges in accurately representing objects in their environment due to the lack of inherent depth information from image sensors, leading to inaccurate bounding box determinations and potential safety hazards.
Innovation Solution
The integration of lidar data with image data to refine 3D bounding boxes using machine learned models, which combine sensor modalities to enhance depth information and improve object representation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If vision-based object detection is used, then object detection speed is improved, but depth information accuracy deteriorates
Solution Approach 1:
The patent combines vision sensor data with lidar sensor data to create a fused 3D bounding box representation. The vision system provides fast 2D detection while the lidar system provides accurate depth information, and their fusion resolves the contradiction between speed and depth accuracy by leveraging the strengths of both sensing modalities
Solution Approach 2:
The patent introduces a 3D bounding box refinement module as an intermediary that takes the 2D bounding box from vision and the point cloud from lidar, then produces an improved 3D bounding box with accurate depth. This intermediary component mediates between the fast but shallow vision detection and the accurate but slower lidar measurement
2Measurement precision
If lidar data is integrated with image data, then object representation accuracy is improved, but system complexity increases
Solution Approach 1:
The patent segments the object representation task into distinct components: 2D bounding box detection from vision, 3D point cloud processing from lidar, and 3D bounding box refinement through fusion. This segmentation allows each component to be optimized independently while maintaining overall system manageability despite the increased complexity of multi-sensor integration
3Reliability
If multi-sensor fusion is implemented, then safety is improved, but computational requirements increase
Solution Approach 1:
The patent performs preliminary 2D object detection using the vision system to identify candidate objects of interest before processing their corresponding 3D point cloud data with lidar. This preliminary action filters out non-object regions, reducing the computational energy required for full 3D processing while maintaining safety through the subsequent fusion refinement step
Data Source
AI summary
Techniques for refining an image-based bounding box with depth data from a lidar sensor to represent an object in an environment of a vehicle are discussed herein. A vehicle computing device can implement a model to determine a three-dimensional bounding box based on image data. The vehicle computing device can implement a same of different model to update the three-dimensional bounding box based at least in part on depth information associated with lidar data. The techniques can also include training one or more machine learned models to determine three-dimensional bounding boxes usable by a vehicle to navigate in the environment.