Weakly-Supervised 3D Object Detection With Multimodal Frustum Alignment

The patent applies local quality by differentiating the detection approach based on object type. For pedestrians and cyclists, it uses density-based clustering to identify regions with concentrated points, while for vehicles, it can use the traditional frustum-aware geometric reasoning. This allows the system to adapt the detection strategy to the local characteristics of each object type, improving overall detection versatility while maintaining precision for each category.

The patent merges 2D image features with 3D point cloud features to create a multimodal detection system. By combining information from multiple modalities (2D RGB images and 3D point clouds), the system increases the effective data points available for detecting pedestrians and cyclists, thereby improving generalization performance and reducing overfitting without excessively increasing system complexity.

The patent applies preliminary action by performing feature alignment and normalization before fusion. It pre-processes the 2D and 3D features to ensure they are in compatible formats and coordinate systems, and establishes correspondence between them in advance. This preliminary preparation prevents error accumulation during the fusion process by ensuring that features are properly aligned and scaled before being combined, thereby maintaining detection accuracy.

The patent replaces the traditional point-wise feature fusion approach with a density-based clustering mechanism. Instead of directly fusing features at corresponding points, it uses density-based clustering to identify regions of interest in the point cloud, then aggregates features within these regions. This substitution allows for effective feature fusion while preserving the resolution differences between 2D and 3D imaging, as the clustering process naturally adapts to the point distribution density rather than requiring exact point-to-point correspondence.