Headset Camera Controller Tracking With Flashing Fiducials

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing augmented reality systems face challenges in accurately tracking the position and orientation of handheld devices, such as controllers, which affects the natural integration of virtual content with real-world imagery, and there is a need for improved localization methods beyond electromagnetic tracking systems.

Innovation Solution

A wearable system that alternates between headset tracking and controller tracking using exposure intervals and fiducial flashes, employing headset and controller cameras, inertial measurement units, and fiducial geometry to determine the pose of the headset and controller, and integrates modular controllers with interchangeable components for enhanced localization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If electromagnetic tracking systems are used for controller localization, then tracking coverage is improved, but system complexity and cost increase

Engineering Contradiction:

Improvetracking accuracyVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent replaces electromagnetic tracking systems with an optical-mechanical system. The controller uses physical fiducial markers (visual features) that are captured by the headset camera, substituting electromagnetic field-based tracking with optical image-based tracking. This reduces system complexity while maintaining tracking accuracy through visual feature recognition and geometric calculation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If continuous high-exposure imaging is used for controller tracking, then fiducial detection accuracy is improved, but motion blur increases

Engineering Contradiction:

Improvefiducial detection accuracyVSAvoidimage clarity

Core Design Contradiction:

Measurement precisionVSShape

Solution Approach 1:

The patent implements periodic flashing of fiducial markers on the controller at specific frequencies. By synchronizing the flash timing with the camera exposure intervals, the system ensures that fiducial markers are illuminated and captured clearly during tracking frames. This periodic action allows for accurate detection while managing exposure duration to prevent motion blur.

Inventive Principle:

Principle #19Periodic action

3Reliability

If fiducial flash frequency is increased for better detection, then tracking reliability is improved, but power consumption increases

Engineering Contradiction:

Improvetracking reliabilityVSAvoidcontroller power consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent uses periodic flashing of fiducial markers at optimized frequencies that balance detection reliability with power consumption. The system determines appropriate flash frequencies based on tracking requirements and synchronizes with camera exposure intervals, ensuring reliable detection while minimizing unnecessary power consumption from continuous illumination.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The patent dynamically adjusts fiducial flash parameters (frequency, duration, intensity) based on tracking conditions and camera exposure settings. By changing these parameters adaptively, the system maintains reliable tracking detection while optimizing power consumption according to actual operational needs rather than using fixed high-frequency flashing.

Inventive Principle:

Principle #35Parameter changes

4Measurement precision

If alternating exposure intervals are used for headset and controller tracking, then tracking precision is improved, but processing time increases

Engineering Contradiction:

Improvepose tracking precisionVSAvoidtracking processing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent implements a periodic alternating pattern between headset tracking frames and controller tracking frames. Each frame type uses appropriate exposure intervals optimized for its specific tracking requirements. This structured periodic alternation allows the system to maintain high precision for both headset and controller tracking while managing processing throughput through regular frame scheduling.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS12487663B2Wearable system with controller localization using headset cameras and controller fiducials

Publication Date: 2025.12.02 MAGIC LEAP INC

AI summary

Wearable systems and method for operation thereof incorporating headset and controller localization using headset cameras and controller fiducials are disclosed. A wearable system may include a headset and a controller. The wearable system may alternate between performing headset tracking and performing controller tracking by repeatedly capturing images using a headset camera of the headset during headset tracking frames and controller tracking frames. The wearable system may cause the headset camera to capture a first exposure image an exposure above a threshold and cause the headset camera to capture a second exposure image having an exposure below the threshold. The wearable system may determine a fiducial interval during which fiducials of the controller are to flash at a fiducial frequency and a fiducial period. The wearable system may cause the fiducials to flash during the fiducial interval in accordance with the fiducial frequency and the fiducial period.