Lightweight Cross-Reality Headset Tracking with DVS and Color Cameras
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Wearable XR systems face challenges with weight, power consumption, and accuracy in tracking objects due to sensor position changes and high power requirements for frequent data acquisition, leading to reduced user enjoyment and realism in XR experiences.
Innovation Solution
A wearable cross reality display system using a combination of a DVS camera and a color camera, with a processor that dynamically processes image data based on detected conditions, employs a calibration routine to maintain accurate stereoscopic depth information and reduces power consumption by selectively activating sensors, enabling low-latency object tracking and head pose estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sensors are activated frequently for data acquisition to maintain high temporal resolution and accurate object tracking, then measurement precision and tracking accuracy are improved, but power consumption increases
Solution Approach 1:
The system implements selective sensor activation where the DVS camera and color camera are activated periodically or on-demand based on detected conditions rather than continuously. The processor monitors for specific events or changes in the environment and only activates sensors when needed, creating a periodic action pattern that maintains tracking accuracy while significantly reducing overall power consumption during stable periods.
Solution Approach 2:
The system dynamically changes operational parameters by adjusting sensor activation states based on detected conditions. When motion is detected or specific events occur, the system transitions from a low-power state to an active sensing state, changing the parameter of sensor operation from inactive to active only when necessary, thereby optimizing the balance between tracking precision and power consumption.
2Measurement precision
If multiple cameras are used to provide overlapping views for accurate depth information and object tracking, then measurement precision is improved, but device weight increases
Solution Approach 1:
The system merges the functionality of multiple cameras (DVS camera and color camera) into a unified processing pipeline where both sensors capture data from overlapping fields of view. The processor integrates information from both cameras to compute stereoscopic depth and track objects, combining their complementary strengths to achieve high measurement precision while sharing the computational load.
Solution Approach 2:
The system employs selective camera activation where not all cameras operate at full capacity continuously. Instead, the processor determines which camera or combination of cameras is needed based on current tracking requirements and environmental conditions, activating only the necessary sensors at any given moment, thus reducing the effective weight burden while maintaining depth accuracy.
3Reliability
If sensors continuously acquire data at high temporal resolution to maintain realism in XR environments, then visual realism is improved, but power consumption increases
Solution Approach 1:
The system implements periodic data acquisition where sensors capture high-resolution data at specific intervals or upon detecting significant events rather than continuously. During periods of stability, the system reduces sampling frequency to conserve power, while maintaining high temporal resolution when changes are detected, thus preserving XR realism while managing power consumption dynamically.
Solution Approach 2:
The system employs self-service mechanisms where the processor autonomously monitors sensor data streams and automatically activates or deactivates sensors based on detected conditions without external intervention. This self-regulating approach ensures that high temporal resolution is maintained only when necessary for realism, allowing the system to serve its own power management needs and optimize the balance between visual fidelity and energy consumption.
Data Source
AI summary
A wearable display system for a cross reality (XR) system may have a dynamic vision sensor (DVS) camera and a color camera. At least one of the cameras may be a plenoptic camera. The wearable display system may dynamically restrict processing of image data from either or both cameras based on detected conditions and XR function being performed. For tracking an object, image information may be processed for patches of a field of view of either or both cameras corresponding to the object. The object may be tracked based on asynchronously acquired events indicating changes within the patches. Stereoscopic or other types of image information may be used when event-based object tacking yields an inadequate quality metric. The tracked object may be a user's hand or a stationary object in the physical world, enabling calculation of the pose of the wearable display system and of the wearer's head.