Metaverse User Localization With Inertial-Visual Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing user localization methods in the metaverse suffer from poor accuracy due to reliance on continuous visual tracking, which fails in environments with weak visual information, leading to unrealistic user displacements and loss of tracking, and do not effectively integrate inertial movements, resulting in unexpected user movements and inaccurate placement in virtual worlds.
Innovation Solution
A method utilizing inertial sensors and cameras to generate sensor and visual data, processed by ML models to extract stable key points, filter out dynamic key points, and integrate sensor data with higher weightage to improve localization accuracy, incorporating loop closure and pose optimization for robust tracking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous visual tracking is used for user localization, then localization can be maintained in most environments, but tracking fails in environments with weak visual information (repeated textures, shimmering light, motion blur)
Solution Approach 1:
The patent combines visual tracking data from cameras with inertial measurement data from sensors (accelerometers, gyroscopes) to create a hybrid localization system. The sensor fusion module integrates these two data sources, allowing the system to maintain reliable tracking even when visual information is weak or unavailable, thus resolving the contradiction between tracking reliability and adaptability to weak visual environments.
Solution Approach 2:
The patent introduces sensor data as an intermediary to bridge the gap between visual tracking failures and accurate user localization. When visual tracking fails due to weak visual information, the sensor data serves as a mediator to provide alternative localization information, ensuring continuous and reliable user position tracking throughout the metaverse environment.
2Measurement precision
If visual tracking is used for user localization, then user position can be determined, but errors are introduced due to sensor bias and distortion in mapping
Solution Approach 1:
The patent implements a feedback mechanism where sensor data continuously corrects and refines visual tracking results. The sensor fusion module processes both visual and inertial data, using feedback from sensor measurements to compensate for visual tracking errors caused by sensor bias and mapping distortion, thereby improving both localization precision and tracking accuracy.
Solution Approach 2:
The patent changes the parameters used for localization by incorporating multiple data sources (visual and inertial) with different characteristics. The system dynamically adjusts the weight and processing parameters of each data source based on environmental conditions, allowing it to maintain high precision and accuracy across diverse metaverse environments by adapting to changing visual and sensor quality.
3Measurement precision
If key points are extracted from visual data for localization, then user position can be mapped, but dynamic objects cause unexpected movements in the virtual world
Solution Approach 1:
The patent extracts and separates reliable tracking information from unreliable visual data by filtering out dynamic objects and spurious key points. The sensor fusion module identifies and removes harmful factors such as movements caused by dynamic objects, extracting only the genuine user position information needed for accurate virtual world mapping, thus preventing unrealistic user movements.
Solution Approach 2:
The patent segments the visual data into reliable and unreliable components by distinguishing between static environmental features and dynamic objects. The system processes visual and sensor data separately, then integrates only the reliable segments that correspond to actual user movements, excluding segments caused by dynamic objects, thereby maintaining localization precision while eliminating harmful effects.
4Device complexity
If visual data alone is used for localization, then processing is simpler, but user dynamism and inertial movements are not preserved
Solution Approach 1:
The patent merges visual data processing with sensor data processing in a unified sensor fusion framework. This integration allows the system to maintain user dynamism and inertial movements accurately while managing complexity through coordinated processing of multiple data sources, achieving reliable user movement representation without excessive computational burden.
Data Source
AI summary
The present disclosure provides a method for intelligent user localization in a metaverse, including: detecting movements of a wearable head gear configured to present virtual content to a user, and generating sensor data and visual data using an inertial sensor and a camera, respectively, mapping the visual data to a virtual world using an image associated with the visual data to localize the user in the virtual world; providing the visual data and the sensor data to a first Machine Learning (ML) model and a second ML model, respectively; extracting a plurality of key points from the visual data and distinguishing stable key points and dynamic key points; and removing visual impacts corresponding to the visual data having a relatively low weightage, and providing a relatively high weightage to the sensor data processed through the second ML model.