Sparse CNN Tiling for Real-Time LIDAR Feature Map Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Standard deep convolutional neural networks require high computational costs for real-time applications due to the need to perform convolutions across entire feature maps, which is inefficient for sparse imagery like LIDAR data in autonomous vehicles, where only specific regions are relevant for object detection and semantic segmentation.
Innovation Solution
The implementation of a novel tiling-based sparse convolutional algorithm that leverages the sparsity of input data by using sparse blocks networks (SBNet) to compute convolutions only on relevant portions of the imagery, identified through binary masks generated based on data density or machine-learned models, thereby reducing unnecessary computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard deep convolutional neural networks perform convolutions across entire feature maps, then comprehensive object detection and semantic segmentation are achieved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the feature map into multiple tiles or regions, processing only non-sparse regions containing actual data points. This segmentation allows the network to skip sparse regions entirely, reducing computational load while maintaining detection accuracy in relevant areas.
Solution Approach 2:
The patent applies different processing strategies to different regions of the feature map based on their sparsity characteristics. Non-sparse regions receive full convolutional processing while sparse regions are skipped or handled with simplified operations, optimizing the trade-off between accuracy and speed.
2Reliability
If convolutions are performed on entire imagery, then complete scene understanding is achieved, but computational resources are wasted on sparse regions with no data
Solution Approach 1:
The patent extracts and identifies sparse regions from the feature map before processing, separating them from non-sparse regions. This extraction allows the system to exclude sparse regions from convolution operations, eliminating wasted computational energy while preserving complete scene understanding through proper handling of boundaries between sparse and non-sparse regions.
Solution Approach 2:
The patent performs convolutions only on the necessary portions of the feature map (non-sparse regions) rather than the entire map. This partial action approach maintains sufficient scene understanding for autonomous driving tasks while significantly reducing computational energy consumption.
3Measurement precision
If standard convolutional networks process LIDAR data, then accurate 3D object detection is achieved, but processing speed is insufficient for real-time autonomous vehicle applications
Solution Approach 1:
The patent segments the 3D LIDAR feature space into spatial tiles and processes only those tiles containing non-sparse data. This segmentation enables parallel processing of relevant regions while skipping empty spaces, thereby increasing processing speed without compromising detection accuracy.
Solution Approach 2:
The patent changes the processing parameters dynamically based on sparsity patterns, adjusting which regions receive full convolutional processing. This parameter adaptation allows the system to maintain high detection accuracy for objects while processing at speeds suitable for real-time autonomous vehicle operations.
Data Source
AI summary
The present disclosure provides systems and methods that apply neural networks such as, for example, convolutional neural networks, to sparse imagery in an improved manner. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can extract one or more relevant portions from imagery, where the relevant portions are less than an entirety of the imagery. The computing system can provide the relevant portions of the imagery to a machine-learned convolutional neural network and receive at least one prediction from the machine-learned convolutional neural network based at least in part on the one or more relevant portions of the imagery. Thus, the computing system can skip performing convolutions over regions of the imagery where the imagery is sparse and/or regions of the imagery that are not relevant to the prediction being sought.