Wearable XR Display Tracking with Event-Based Camera Patches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing wearable XR systems face challenges with weight, power consumption, and accuracy in capturing and processing high-resolution, high-framerate visual data, leading to reduced user enjoyment and system utility due to fatigue, battery life, and inaccurate virtual object placement relative to physical objects.
Innovation Solution
A wearable cross reality display system utilizing a combination of grayscale and RGB cameras, event-based data acquisition, and patch tracking to reduce power consumption while maintaining high temporal resolution, coupled with a calibration routine to adjust for sensor shifts and enhance stereoscopic depth accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-resolution, high-framerate visual data is captured and processed, then temporal resolution is improved, but power consumption increases
Solution Approach 1:
The system uses event-based data acquisition where cameras capture visual data only when changes occur, rather than continuously at high framerate. This periodic action triggered by events maintains high temporal resolution for dynamic elements while dramatically reducing overall power consumption during static periods.
Solution Approach 2:
The system dynamically adjusts data acquisition based on scene activity. When motion or changes are detected, the system increases sampling rate and processing intensity; when the scene is static, it reduces activity. This dynamic adaptation allows high temporal resolution when needed while conserving power during calm periods.
2Measurement precision
If high-resolution, high-framerate visual data is captured and processed, then temporal resolution is improved, but device weight increases
Solution Approach 1:
By using event-triggered periodic action rather than continuous high-framerate capture, the system reduces the computational burden and processing power required. This allows achieving high temporal resolution with less sophisticated (and lighter) hardware components.
Solution Approach 2:
The system extracts only the necessary visual information through event-based sampling, capturing only changes rather than complete high-resolution frames continuously. This extraction approach reduces processing requirements and enables lighter device design while maintaining temporal resolution for relevant events.
3Device complexity
If sensors are used without calibration, then device complexity is reduced, but measurement precision deteriorates
Solution Approach 1:
The system performs calibration routines in advance before actual operation. By pre-calibrating sensor shifts and stereoscopic parameters, the system ensures high depth accuracy during use without adding complexity to the operational workflow. The calibration is a preliminary setup step rather than an ongoing complexity.
Data Source
AI summary
A wearable display system for a cross reality (XR) system may have a dynamic vision sensor (DVS) camera and a color camera. At least one of the cameras may be a plenoptic camera. The wearable display system may dynamically restrict processing of image data from either or both cameras based on detected conditions and XR function being performed. For tracking an object, image information may be processed for patches of a field of view of either or both cameras corresponding to the object. The object may be tracked based on asynchronously acquired events indicating changes within the patches. Stereoscopic or other types of image information may be used when event-based object tacking yields an inadequate quality metric. The tracked object may be a user's hand or a stationary object in the physical world, enabling calculation of the pose of the wearable display system and of the wearer's head.