Bird's-Eye View Mapping From FPV Images With Monocular Depth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous navigation systems face challenges in accurately and reliably projecting first-person view (FPV) modalities to bird's eye view (BEV) maps due to the reliance on costly LIDAR sensors and end-to-end supervised learning methods that require extensive training and custom datasets, making them cumbersome and expensive, especially when adapting to changes in object classes.
Innovation Solution
A method using machine-learning models to generate BEV maps from FPV images by extracting and concatenating feature maps, employing transformer architectures for geometric correspondence, and incorporating monocular depth estimation to create residual BEV feature maps, enabling zero-shot projection of various modalities without specific training for each modality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LIDAR sensors are used to obtain depth information, then measurement precision is improved, but device complexity and cost increase
Solution Approach 1:
The patent replaces LIDAR sensors with a computer vision-based depth estimation system. Instead of using mechanical/optical sensors to directly measure depth, the system uses machine learning models trained on image data to infer depth information, thereby eliminating the need for LIDAR hardware while maintaining depth measurement capabilities
Solution Approach 2:
The patent creates synthetic training data by rendering 3D scenes with ground truth depth information. This synthetic data serves as a copy or representation of real-world depth relationships, allowing the machine learning model to learn depth estimation without requiring physical depth sensors during training or operation
2Measurement precision
If end-to-end supervised learning is used to estimate depth information, then measurement precision is improved, but device complexity and training cost increase
Solution Approach 1:
The patent pre-computes and stores depth information from 3D scene renderings during the training phase. This preliminary action allows the machine learning model to learn accurate depth relationships in advance, so that during actual operation, the model can quickly infer depth without requiring complex real-time computation or retraining
Solution Approach 2:
The patent divides the depth estimation task into separate components: a 3D scene rendering pipeline that generates training data with ground truth depth, and a machine learning model that learns to map images to depth. This segmentation allows each component to be optimized independently, reducing overall system complexity while maintaining precision
3Measurement precision
If modality-specific training is performed for each projected modality, then measurement precision is improved, but adaptability deteriorates
Solution Approach 1:
The patent creates a universal machine learning model that can handle multiple modalities (RGB images, depth maps, segmentation masks, optical flow) through a single unified architecture. The model learns to project any input modality to BEV maps using the same underlying framework, eliminating the need for separate training for each modality while maintaining high accuracy for all supported modalities
Data Source
AI summary
Methods and systems described herein generate a bird-eye view (BEV) map from a first-person view (FPV) image of a scene using trained machine-learning models. The methods include: generating a modal image, corresponding to the FPV image, that is representative of a feature of the FPV image; extracting, from the FPV image, a first set of feature maps (FM) with a first model, and from the modal image, a second set of FM with a second model; concatenating the first set of FM with the second set of FM to generate a set of tensors; generating a set of BEV FM with a third model that maps the set of tensors to the set of BEV FM; and decoding the set of BEV FM with a fourth model to generate the BEV map with the feature projected thereon.