SPAD Lidar Instance Segmentation Using CNN and PointNet
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network-based approaches for panoptic segmentation, such as DeepLab, MASK-RCNN, Point-RCNN, and Voxelnet, fail to fully utilize the capabilities of SPAD Lidar systems by focusing either on image or point cloud data, leading to incomplete utilization of SPAD Lidar's potential for accurate instance segmentation and 3D object detection.
Innovation Solution
A method combining a two-dimensional convolutional neural network (CNN) with a U-Net architecture and a PointNet model to process SPAD Lidar data, where the CNN filters out background pixels and the PointNet model generates pixel-level instance segmentation, leveraging the SPAD Lidar's ability to capture multiple return signals, ambient scene illumination, and surface reflectance for improved object detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing neural network approaches (DeepLab, MASK-RCNN, Point-RCNN, Voxelnet) are used for panoptic segmentation, then either image or point cloud data is processed, but the full capacity of SPAD Lidar systems cannot be utilized
Solution Approach 1:
The patent combines image data processing (2D CNN) and point cloud data processing (PointNet) into a unified panoptic segmentation framework. This merging allows the system to simultaneously utilize both the spatial information from images and the depth information from point clouds, fully leveraging the SPAD Lidar system's capabilities while achieving accurate instance segmentation.
Solution Approach 2:
The patent creates a universal framework that can process multiple types of data (image data and point cloud data) through a single panoptic segmentation system. The system universally handles both 2D image information and 3D point cloud information, making it adaptable to the multi-functional capabilities of SPAD Lidar systems.
2Device complexity
If only image data is processed using 2D CNN, then computational complexity is reduced, but 3D spatial information from SPAD Lidar is lost
Solution Approach 1:
The patent segments the processing pipeline into two specialized components: a 2D CNN branch for processing image data and a PointNet branch for processing point cloud data. Each segment handles its specific data type with appropriate computational complexity, while both segments contribute to the final panoptic segmentation result, preserving 3D spatial information without overwhelming computational demands.
3Measurement precision
If only point cloud data is processed using PointNet, then 3D object detection capability is improved, but detailed spatial context from image data is lost
Solution Approach 1:
The patent merges the outputs of the 2D CNN (providing detailed spatial context from image data) and the PointNet (providing 3D object detection capability from point cloud data) into a unified panoptic segmentation framework. This combination ensures that both the detailed spatial context and the 3D detection capabilities are preserved and integrated.
4Productivity
If traditional segmentation approaches are used, then processing speed is maintained, but panoptic segmentation accuracy is insufficient
Solution Approach 1:
The patent performs preliminary processing of image data and point cloud data through separate specialized networks (2D CNN and PointNet) before combining them for panoptic segmentation. This preliminary action allows each data type to be optimally processed independently, improving the overall accuracy of the final segmentation while maintaining efficient processing through parallel computation.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The method achieves accurate 2D instance segmentation and 3D bounding box estimation by effectively utilizing both ambient and point cloud data, enhancing the performance of SPAD Lidar systems in object detection tasks.
Implementation Method 1
capture surface reflectance by measuring a strength of a detected laser pulse
Implementation Method 2
single-photon avalanche diode (SPAD) light detection and ranging (Lidar)
Implementation Method 3
capture infrared (IR) light to capture ambient scene illumination
Implementation Method 4
photon detector of an SPAD Lidar sensor
Data Source
AI summary
A computerized method for instance segmentation using single-photon avalanche diode (SPAD) light detection and ranging (Lidar) includes obtaining sensor image data from photon detector of an SPAD Lidar sensor, supplying the obtained sensor image data to a two-dimensional convolutional neural network (CNN) to generate a background mask by identifying target objects in the obtained sensor image data and filtering out background pixels that do not belong to the identified target objects, and supplying point cloud data of the identified target objects to a PointNet model to generate a pixel level instance segmentation output of the identified target objects.