Learned Occlusion Modeling for XR Feature-Point Localization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extended reality (XR) systems face challenges in accurately modeling occlusions, leading to incorrect tracking of feature points, excessive computing resource usage, and loss of tracking due to inefficient occlusion prediction, which affects battery life and portability.
Innovation Solution
Implement learned occlusion modeling using an occlusion prediction engine with an embedder and visibility classifier to predict feature point visibility based on historical observations, enhancing localization accuracy and robustness in feature-poor environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional occlusion modeling is used, then feature point tracking can be performed, but localization accuracy deteriorates due to incorrect tracking
Solution Approach 1:
The patent replaces traditional geometric occlusion modeling with a machine learning-based occlusion prediction model. The neural network takes camera pose and map data as input and directly predicts occlusion probabilities for feature points, substituting complex geometric calculations with a learned probabilistic model that better captures real-world occlusion patterns.
Solution Approach 2:
The system uses observation history from previous frames as feedback to update the occlusion prediction model. By continuously refining predictions based on actual observations of feature point visibility, the system improves its ability to accurately model occlusions over time, leading to better localization accuracy.
2Measurement precision
If comprehensive feature point tracking is performed, then localization can be achieved, but computing resource usage increases excessively
Solution Approach 1:
Instead of processing all feature points equally, the system uses occlusion predictions to selectively process only the most relevant feature points. Feature points with high occlusion probability are prioritized for processing, while those with low probability are skipped or processed with reduced computational effort, achieving partial action that maintains accuracy while reducing resource usage.
Solution Approach 2:
The patent applies different processing qualities to different feature points based on their occlusion characteristics. High-confidence visible features receive full processing, while occluded or ambiguous features receive reduced processing, creating a non-uniform processing strategy that optimizes the balance between accuracy and computational efficiency.
3Speed
If powerful processors are used for feature analysis, then processing speed improves, but battery life decreases
Solution Approach 1:
The system performs partial feature analysis by using occlusion predictions to filter and prioritize feature points. Instead of processing all detected features through the full pipeline, the system processes only the subset most likely to be visible and relevant, reducing the overall computational load on the processor and thereby extending battery life while maintaining acceptable processing speed.
4Device complexity
If occlusion prediction is performed without history information, then processing is simpler, but tracking stability deteriorates
Solution Approach 1:
The system performs preliminary processing by storing and indexing observation history from previous frames before generating current occlusion predictions. This pre-computed historical data is then queried during prediction, allowing the system to leverage past information without adding complex real-time computation, thereby improving tracking stability while keeping the prediction model itself relatively simple.
Data Source
AI summary
Techniques and systems are provided for image generation. For instance, a process can include receiving a location for a device and a map of feature points in the environment; predicting whether one or more feature points of the map of feature points are visible by at least one sensor from the location based on history information related to at least a plurality of feature points included in the map of feature points, wherein the history information indicates previous locations of the device and feature points of the map visible by the at least one sensor from the previous locations; and determining a location of the device in the environment based on the one or more feature points predicted to be visible by the at least one sensor from the location.