Monocular 3D Object Detection Using Voxelized BEV Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D object detection methods are domain-specific, complex, and challenging to implement, especially when using monocular RGB images, as they lack explicit scene geometry and scale information, and often require custom and complex neural network architectures.
Innovation Solution
A method and system that utilize multi-view inputs for both training and inference, constructing a voxel representation of 3D space, and employing a modular neural network design with end-to-end training, allowing for domain-agnostic object detection and scene understanding, using a unified approach for both indoor and outdoor scenes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If domain-specific 3D object detection methods are used for indoor or outdoor scenes, then detection accuracy for specific domains is improved, but device complexity and implementation difficulty increase due to custom neural network architectures
Solution Approach 1:
The patent applies universality by designing a domain-agnostic 3D object detection method that can handle both indoor and outdoor scenes using the same neural network architecture. The method processes monocular images through a unified approach that constructs 3D scene geometry and detects objects without requiring domain-specific custom architectures, thereby reducing implementation complexity while maintaining versatility across different environments.
2Ease of manufacture
If monocular RGB images are used for 3D object detection, then data acquisition is simplified and cost is reduced, but measurement precision deteriorates due to lack of explicit scene geometry and scale information
Solution Approach 1:
The patent applies dimensionality change by transforming 2D monocular image data into 3D scene representations through geometric construction. The method recovers scene geometry, camera poses, and object 3D positions by lifting 2D image features into 3D space using epipolar geometry and perspective projection models, thereby obtaining explicit 3D measurement information from inherently 2D input data.
Solution Approach 2:
The patent uses an intermediary approach by introducing a neural network-based scene understanding module that acts as a bridge between monocular images and 3D object detection. This intermediary constructs intermediate representations including scene geometry, camera poses, and spatial relationships, which then enable accurate 3D object detection despite the limitations of monocular input.
3Measurement precision
If multi-view inputs are used for inference only while training uses different data, then inference performance may improve, but loss of information occurs during the domain gap between training and inference stages
Solution Approach 1:
The patent applies preliminary action by performing domain adaptation during the training phase. The method trains the neural network on monocular images with constructed 3D scene geometry and camera poses, preparing the model in advance to handle the specific characteristics of inference data. This preliminary training ensures the network is adapted to the same domain conditions during inference, eliminating domain gap issues without requiring separate multi-view training data.
Data Source
AI summary
A method of 3D object detection using an object detection neural network includes: receiving one or more monocular images; extracting 2D feature maps from each one of the one or more monocular images by passing the one or more monocular images through a 2D feature extracting part of the object detection neural network, generating an averaged 3D voxel volume based on the 2D feature maps, extracting a 2D representation of 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through an encoder of a 3D feature extracting part of the object detection neural network, and performing 3D object detection as 2D object detection in a Bird's Eye View (BEV) plane, the 2D object detection in the BEV plane being performed by passing the 2D representation of 3D feature maps through thane outdoor object detecting part of the object detection neural network.