Multi-Camera Depth Mapping for Lightweight XR Wearables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing wearable cross reality (XR) systems face challenges with weight, power consumption, and accuracy in capturing and processing depth information due to sensor shifts and high power requirements, leading to reduced user enjoyment and realism in XR experiences.
Innovation Solution
A wearable XR system utilizing a combination of cameras with global and rolling shutters, along with a processor, performs compensation and size reduction routines to adjust images, creates a world model using stereoscopic depth information, and selectively activates sensors to reduce power consumption and maintain accurate depth perception.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple sensors are used to capture depth information, then measurement precision is improved, but device complexity and weight increase
Solution Approach 1:
The patent combines data from multiple sensors (time of flight sensor, stereoscopic cameras, and structure from motion algorithms) into a unified depth map through sensor fusion. This merging approach maintains high measurement precision while reducing the need for separate dedicated depth sensors, thereby lowering device complexity and weight.
Solution Approach 2:
The system uses cameras with dual purposes: they capture both 2D image data and depth information through stereoscopic vision and structure from motion algorithms. This multi-functionality eliminates the need for separate depth sensors, reducing device complexity while maintaining depth measurement precision.
2Measurement precision
If high-resolution depth sensing is implemented, then measurement precision is improved, but power consumption increases
Solution Approach 1:
The system selectively activates sensors based on operational needs. The time of flight sensor and stereoscopic cameras are activated only when depth information is required, rather than continuously operating at full resolution. This partial action approach maintains measurement precision when needed while significantly reducing overall power consumption.
Solution Approach 2:
The system implements periodic depth sensing rather than continuous high-resolution depth capture. Depth maps are generated at specific intervals or triggered by motion detection, allowing the system to maintain accurate depth information when required while reducing power consumption during static periods.
3Measurement precision
If sensor activation is continuous, then measurement precision is maintained, but power consumption and battery life are worsened
Solution Approach 1:
The system dynamically adjusts sensor activation based on real-time conditions. Sensors are activated when motion is detected, when depth information is required for rendering, or when the device transitions between states. This dynamic approach maintains measurement precision during active periods while extending battery life through reduced power consumption during inactive periods.
Solution Approach 2:
The system uses feedback from motion sensors, tracking data, and rendering requirements to control sensor activation. When the system detects that depth information is not currently needed (e.g., during static viewing or when virtual objects are not interacting with real objects), it reduces sensor activity, thereby extending battery life while maintaining measurement precision when feedback indicates depth sensing is required.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The system achieves lightweight, low-power operation with enhanced accuracy in capturing and processing depth information, enabling realistic interaction of virtual objects with the physical world, extending battery life without compromising user experience.
Implementation Method 1
perform a compensation routine to adjust images acquired using the second camera for rolling shutter image distortion
Implementation Method 2
create a world model using, in part, depth information stereoscopically determined from images acquired using the first camera and the adjusted images
Data Source
AI summary
A wearable display system including multiple cameras and a processor is disclosed. A greyscale camera and a color camera can be arranged to provide a central view field associated with both cameras and a peripheral view field associated with one of the two cameras. One or more of the two cameras may be a plenoptic camera. The wearable display system may acquire light field information using the at least one plenoptic camera and create a world model using the first light field information and first depth information stereoscopically determined from images acquired by the greyscale camera and the color camera. The wearable display system can track head pose using the at least one plenoptic camera and the world model. The wearable display system can track objects in the central view field and the peripheral view fields using the one or two plenoptic cameras, when the objects satisfy a depth criterion.