XR Ray-World Intersection Using Depth Maps on Mobile Glasses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR devices face computational challenges in generating 3D representations of the user's environment, particularly in real-time, due to the limitations of existing ray-casting approaches and the need for specialized sensors, which are not suitable for small mobile devices like smart glasses.
Innovation Solution
Leveraging sparse depth data from VIO and hand tracking subsystems, combined with lightweight machine learning models, to generate depth maps and perform computationally efficient ray-world intersection modeling, suitable for real-time execution on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing ray-casting approaches are used to generate 3D representations, then measurement precision is improved, but device complexity and computational requirements increase beyond what small mobile devices can handle
Solution Approach 1:
The patent creates a simplified voxel-based copy of the 3D environment from video frames, replacing complex continuous 3D models with discrete voxel representations. This copying approach maintains sufficient measurement precision for ray-world intersection while dramatically reducing computational complexity and memory requirements for mobile devices.
Solution Approach 2:
The patent transforms the problem from continuous 3D space to discrete voxel space by changing the parameter representation. Instead of working with continuous coordinates and complex geometric models, the system uses discrete voxel grids with simplified occupancy parameters, making ray-casting computations feasible on mobile hardware.
2Measurement precision
If specialized sensors are added to improve ray-casting capability, then measurement precision is improved, but device complexity and size increase
Solution Approach 1:
The patent makes the existing camera system serve dual purposes: capturing both video frames for display and depth information for ray-casting. By extracting depth maps from monocular video frames using machine learning models, the system eliminates the need for specialized depth sensors while maintaining functional capability.
Solution Approach 2:
The patent enables the camera subsystem to perform multiple functions simultaneously: visual display capture and 3D environmental mapping for ray-casting. This multi-functionality approach allows the same hardware to support both AR rendering and spatial understanding without adding dedicated sensors.
3Productivity
If computationally intensive 3D modeling is performed in real-time, then productivity is improved, but use of energy increases beyond mobile device capabilities
Solution Approach 1:
The patent segments the 3D environment into discrete voxels and processes only relevant regions for ray-casting operations. This segmentation allows the system to focus computational energy on specific areas of interest rather than processing the entire 3D space, enabling real-time performance within mobile energy constraints.
Solution Approach 2:
The patent uses lightweight, temporarily generated depth maps and voxel representations instead of maintaining persistent complex 3D models. These simplified representations are created on-demand for each frame, requiring minimal energy for generation and disposal, enabling real-time operation on mobile devices.
Data Source
AI summary
A system for world-ray intersection modeling has a memory storing instructions that, when executed by a processor, configure the system to perform operations. Video data comprising a video frame is obtained. Depth map data is obtained, comprising, for each pixel of a plurality of pixels of the video frame, a depth value of an object visible at a pixel location of the pixel. Ray data is obtained, representative of a ray in three-dimensional space. The depth map data and the ray data are processed to generate intersection data comprising a three dimensional location of an intersection of the ray with the object visible at the pixel location. The intersection can be determined using a voxel representation of the depth map data or by stepwise traversal of the ray. A trajectory defined by multiple intersections over time can be smoothed to present virtual content moving in a realistic fashion.