Monocular Depth and Semantic Localization for GPS-Denied Delivery Drones
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the absolute location of unmanned aerial vehicles (UAVs) rely heavily on Global Navigation Satellite Systems (GNSS), which can become unreliable in certain situations, necessitating a backup or complementary modality for accurate localization.
Innovation Solution
A machine-learned model is employed to produce semantically labeled three-dimensional point clouds that can be registered against pre-existing point clouds to derive the UAV's absolute position and orientation, providing a backup to GPS by aligning semantic labels with reference depth data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If GNSS is used for UAV localization, then absolute position can be determined, but reliability deteriorates in GPS-denied environments
Solution Approach 1:
The system performs preliminary actions by capturing multiple images and generating depth maps and semantic segmentations before GNSS failure occurs. The depth map generation unit creates depth information from images captured before localization failure, and the semantic segmentation unit prepares semantic labels in advance. This allows the UAV to determine its position using pre-processed visual data when GNSS becomes unavailable.
Solution Approach 2:
The patent introduces visual data processing as an intermediary system between the UAV and the environment for localization. The depth map generation unit and semantic segmentation unit act as intermediaries that transform images into structured depth and semantic information. This intermediary visual processing system enables the UAV to localize itself through image-based features when direct GNSS positioning fails.
2Measurement precision
If machine learning models process 2D images to produce depth and semantic information, then localization accuracy improves, but device complexity increases
Solution Approach 1:
The system segments the image processing task into distinct functional units: a depth map generation unit that extracts depth information, and a semantic segmentation unit that identifies object categories. This segmentation of processing functions allows each unit to specialize in specific tasks, improving localization precision through dedicated processing while organizing complexity into manageable, modular components.
Solution Approach 2:
The machine learning models perform multiple functions simultaneously - generating depth maps for spatial understanding and semantic segmentations for object identification from the same input images. This multi-functionality allows the system to extract both geometric and semantic information universally from visual data, improving localization accuracy without requiring separate processing systems for each type of information.
Data Source
AI summary
A method includes receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV. The method further includes applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the semantic image comprises one or more semantic labels. The method additionally includes retrieving reference depth data representative of the environment, wherein the reference depth data includes reference semantic labels. The method also includes aligning the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.