Cross-Reality Localization with Persistent Maps and Neural Feature Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cross reality (XR) systems face challenges in efficiently and accurately localizing XR devices in large and very large scale environments, such as neighborhoods, cities, or the globe, due to computational limitations and the need for substantial processing time and resources, which affects the realism and immersion of XR experiences.
Innovation Solution
The XR system employs persistent spatial information represented by persistent maps stored in a remote storage medium, utilizing neural networks to efficiently match and transform feature descriptors across devices, enabling quick and accurate localization of XR devices by reducing computational demands and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional XR systems process images locally to build environment representations, then localization accuracy is maintained, but processing time and computational resources increase substantially
Solution Approach 1:
The patent extracts the computationally intensive image processing and localization tasks from the local XR device and relocates them to a remote server. The XR device captures images and transmits them to the server, which performs feature extraction, descriptor generation, and map matching. This extraction of heavy computation from the endpoint device resolves the contradiction by maintaining localization accuracy through server-side processing while dramatically reducing local processing time and resource usage.
Solution Approach 2:
The patent introduces a remote server as an intermediary between the XR device and the localization service. This intermediary handles the computationally demanding tasks of processing images, extracting features, generating descriptors, and matching them against persistent maps. The server acts as a mediator that receives raw image data from devices, performs complex computations, and returns localization results, thereby resolving the time-resource contradiction while preserving accuracy.
2Productivity
If XR systems use persistent maps stored remotely with neural networks for feature matching, then computational demands and latency are reduced, but system complexity increases
Solution Approach 1:
The patent uses persistent maps that are stored remotely and copied or accessed by multiple XR devices as needed. Instead of each device maintaining its own complete environment representation, the system creates a centralized copy of the persistent map data on the server. This copying approach enables fast localization across multiple devices while centralizing the complexity of map management and neural network processing on the server side.
3Measurement precision
If feature descriptors are matched across multiple devices using remote storage, then localization accuracy improves, but network bandwidth and data transmission requirements increase
Solution Approach 1:
The patent performs preliminary processing of images by extracting features and generating descriptors on the server side before matching them against persistent maps. This preliminary action reduces the amount of raw data that needs to be transmitted between devices and the server. Instead of transmitting complete high-resolution images for processing, the system pre-processes images server-side and transmits only the essential feature data, thereby maintaining localization accuracy while reducing network bandwidth requirements.
Data Source
AI summary
A cross reality system enables any of multiple devices to efficiently and accurately access previously persisted maps of very large scale environments and render virtual content specified in relation to those maps. The cross reality system may quickly determine whether a 2D set of features derived from images acquired with a portable device match a set of 3D features of an environment map and, if so, determine the relative pose of the feature sets. The pose may be used in quickly and accurately localizing the portable device to the environment map. Pairs of features in the 2D and 3D features sets may be identified based on matching feature descriptors and may be scored in a neural network trained to assess the quality of the match. Poses may be identified based on subsets of the matching features weighted towards pairs of features with high quality.