Multi-Camera SLAM for HMD Pose Tracking and FOV Overlap
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Relative pose estimation between a head-mounted display (HMD) and a second device is challenging, especially in low light environments, sparse visual features, no active illumination, non-guaranteed line-of-sight, differing application processors, or temporary device failures, making it difficult to track and illustrate the field-of-view overlap.
Innovation Solution
A system with a main device (HMD) and auxiliary device, each equipped with cameras of different fields of view, uses sensor fusion and image processing to calculate a relative pose, identify occluded areas, and project image frames to indicate overlap, even in challenging conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional single-camera pose estimation is used, then the system is simple, but it fails in low light, sparse visual features, and non-guaranteed line-of-sight conditions
Solution Approach 1:
The system divides the pose estimation task into two independent streams: a wide-FOV camera stream for robust localization and mapping in challenging environments, and a narrow-FOV camera stream for detailed region observation. This segmentation allows each camera to specialize in its strength while the system benefits from their combination.
Solution Approach 2:
The patent merges data from multiple cameras with different FOVs by constructing a unified map from wide-FOV images and then projecting narrow-FOV observations onto this map. The sensor fusion algorithm combines information from both camera streams to achieve reliable pose estimation that leverages the complementary strengths of each camera type.
2Area of stationary object
If wide-FOV camera is used for HMD, then more scene coverage is achieved, but occluded portions cannot be seen directly
Solution Approach 1:
The narrow-FOV camera acts as an intermediary to capture information from occluded regions that the wide-FOV camera cannot directly observe. The system uses the narrow-FOV camera's line-of-sight to peer around obstacles and capture hidden areas, then projects this information onto the wide-FOV map to compensate for occlusions.
Solution Approach 2:
The system adds a temporal dimension to the imaging problem by using sequential narrow-FOV observations to fill in occluded regions over time. Instead of requiring simultaneous multi-view capture, the system accumulates information from the narrow-FOV camera as it moves, progressively revealing occluded areas through time-based dimensionality.
3Measurement precision
If multiple sensors are fused for localization, then accuracy improves, but computational requirements increase
Solution Approach 1:
The computational workload is segmented between two devices: the HMD performs lightweight sensor fusion using its wide-FOV camera and inertial sensors for basic localization, while a separate auxiliary device handles the computationally intensive tasks of map construction and narrow-FOV projection. This segmentation distributes processing demands and reduces the power consumption burden on the HMD.
Data Source
AI summary
In one or more embodiments, instructions that, when executed by a processor, cause the processor to: locate, based on a sensor fusion of a second camera and a second sensor, a first compute device in a map of a 3D scene to define a first device location; calculate, based on the map of the 3D scene and a first sensor, a relative pose of a second compute device with respect to a first compute device location; determine, based on the relative pose, a region of overlap between a FOV of the first camera and a FOV of the second camera; identify, based on the region of overlap, an occluded portion of the second FOV; and send a signal to cause the display to project a plurality of image frames within the second FOV and to reproject the visible portion of the first FOV.