Mixed Reality Device Localization via Keyframe Guidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In mixed reality (MR) systems, users face challenges with relocalization when the environment is unknown to developers, leading to failed localization and poor user experience due to unknown or indiscriminative areas in the environment.
Innovation Solution
A method and system for user device localization in MR systems, which involves obtaining keyframes from the environment, displaying them to the user, and using these keyframes to guide the user in capturing images for accurate sensor position estimation and relocalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional relocalization solutions perform 2D/3D matching or compute whole-image similarity measures, then localization can be achieved, but the user may fail to capture discriminative areas resulting in relocalization failure
Solution Approach 1:
The system performs preliminary action by displaying pre-selected keyframes to the user before the actual localization process. These keyframes show discriminative areas that the user should capture, preparing the user in advance to take the correct action (capturing the right areas) which resolves the contradiction between reliability and ease of operation
Solution Approach 2:
The keyframe acts as an intermediary between the system's localization algorithm and the user. It translates the technical requirement for discriminative features into a visual guide that the user can understand and follow, mediating between the complex 2D/3D matching process and simple user capture actions
2Measurement precision
If the system waits for user to capture images of discriminative areas, then accurate localization can be achieved, but relocalization time increases
Solution Approach 1:
The system identifies and displays the most discriminative keyframes in advance, so the user knows exactly what to capture. This preliminary identification of optimal capture targets reduces the time needed for trial-and-error capturing while ensuring high localization accuracy is achieved on the first attempt
Solution Approach 2:
The system serves itself by automatically selecting and presenting the keyframes that will lead to successful localization. The user simply follows the visual guidance without needing to understand or search for discriminative areas, making the process both fast and accurate
3Adaptability or versatility
If the MR application is used in unknown environments, then versatility is improved, but relocalization reliability deteriorates due to unknown discriminative areas
Solution Approach 1:
Before localization begins in an unknown environment, the system performs preliminary exploration to identify discriminative areas and create keyframes. This advance preparation ensures that even in previously unknown environments, the system can guide the user to capture reliable features, maintaining both versatility and reliability
Solution Approach 2:
The keyframe serves as an intermediary that adapts to any environment. Whether the environment is known or unknown to the system, the keyframe extraction process creates environment-specific visual guides that mediate between the generic MR application and the specific environmental features, ensuring reliable localization across diverse settings
Data Source
AI summary
Localization of a user device in a mixed reality environment in which the user device obtains at least one keyframe from a server, which can reside on the user device, displays at least one of the keyframes on a screen, captures by a camera an image of the environment, and obtains a localization result based on at least one feature of at least one keyframe and the image.