Semantic Segmentation for Localization on Windowed Moving Platforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for localizing electronic devices in 3D environments, particularly on moving platforms like buses, trains, and planes, lack efficiency and accuracy due to unreliable motion sensor data and confusion from external environments visible through windows.
Innovation Solution
Utilizing semantic segmentation to generate keyframes with labeled environmental features, projecting tracking features into these keyframes to identify and exclude outliers, and using different frame rates for semantic and tracking data to improve localization accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If motion sensor data is used for device localization on moving platforms, then localization can be performed, but the accuracy and reliability deteriorate due to unreliable motion data and confusion from external environments
Solution Approach 1:
The patent segments tracking features into two categories: inlier features (from the moving platform environment) and outlier features (from external stationary environment). By separating these feature sets and processing them differently, the system resolves the contradiction between using motion sensor data and maintaining localization accuracy on moving platforms.
Solution Approach 2:
The patent extracts and removes outlier tracking features that correspond to the external stationary environment (visible through windows) from the set of tracking features used for localization. This extraction process eliminates the harmful influence of external environment confusion, improving both reliability and precision of localization.
2Quantity of substance
If all tracking feature points are used for localization, then more data is available for tracking, but accuracy deteriorates due to inclusion of outlier points from external environment
Solution Approach 1:
The patent applies extraction by identifying and removing outlier tracking features that correspond to external environment objects (buildings, trees, etc.) from the total set of tracking features. This is achieved by comparing feature positions across frames and using semantic information to distinguish between moving platform features and external features, thereby improving precision while maintaining an adequate quantity of inlier features.
Solution Approach 2:
The patent applies local quality by treating different tracking features differently based on their semantic classification. Inlier features (from the moving platform) are processed with one set of rules while outlier features (from external environment) are processed with different rules or excluded entirely, optimizing the contribution of each feature type to localization accuracy.
3Measurement precision
If semantic segmentation is performed at high frame rate to match tracking data, then semantic labels are more accurate for tracking features, but computational complexity and processing time increase
Solution Approach 1:
The patent applies periodic action by performing semantic segmentation at a lower frame rate than tracking data acquisition. Semantic keyframes are generated periodically (e.g., every N tracking frames), and these semantic labels are then reused for multiple subsequent tracking feature classifications, reducing computational complexity while maintaining adequate semantic labeling precision.
Solution Approach 2:
The patent applies copying by reusing semantic labels from previously generated semantic keyframes for multiple tracking feature classification operations. Instead of performing semantic segmentation for every tracking frame, the system copies and applies semantic information from earlier keyframes, significantly reducing processing complexity while maintaining labeling accuracy.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that localize (e.g., determine a pose) of a device in a 3D environment based on sensor data and semantic segmentation information. Some implementations provide device localization on moving platforms (e.g., trains, buses, cars, etc.) based on camera images (i.e., vision). Since motion (i.e., IMU) data may not be reliable in such moving environments, image and/or other sensor data may be more heavily relied upon than in other circumstances. Some implementations improve the usability of vision-based tracking features points. This may involve identifying and removing outlier tracking feature points based on semantics. For example, tracking features points corresponding to the outside environment, which is not moving with the moving platform, may be excluded based on semantic information identifying that they are not part of the moving platform (e.g., that they are instead seen through a window, not trackable, etc.).