XR Environment Modeling with Depth Sensing and Selective Fidelity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing extended reality (XR) technologies struggle to accurately model and communicate the local physical environment, particularly in enabling realistic interactions and accurate perception of real-world objects and environments.
Innovation Solution
The use of external environmental cameras with depth sensing capabilities to create textured maps of local environments, combined with machine learning computer vision models to identify and analyze real-world objects, and the integration of avatars for enhanced communication and interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional XR technologies are used to model physical environments, then basic virtual reality functionality is achieved, but accurate modeling and realistic interaction with real-world objects is not possible
Solution Approach 1:
The patent merges multiple camera systems (first and second environment cameras positioned at different locations) with depth sensing capabilities to create a unified accurate model of the physical environment. This combination enables precise spatial understanding and accurate object identification while maintaining manageable system architecture through modular integration of existing XR components.
2Manufacturing precision
If detailed digital models of local environments are created using multiple cameras and depth sensing, then accurate rendering of real-world objects is achieved, but computational requirements increase
Solution Approach 1:
The patent segments the environment modeling task into distinct modules: first environment camera captures initial data, second environment camera captures additional data from different location, depth sensing system processes spatial information, and machine learning models identify objects. This segmentation allows each component to process information independently and efficiently, reducing overall computational burden while maintaining high rendering accuracy.
Solution Approach 2:
The system creates digital copies of the physical environment and objects through photogrammetry and 3D reconstruction. These digital models are then rendered and used for communication purposes, eliminating the need for continuous high-computational processing of real-time sensor data while maintaining accurate representation of the environment.
3Adaptability or versatility
If avatars are integrated for communication in XR environments, then realistic interactions are enabled, but device complexity increases
Solution Approach 1:
The patent introduces avatars as intermediary representations that mediate communication between users in XR environments. Instead of requiring direct complex multi-user tracking and rendering systems, the avatars serve as simplified mediators that convey user presence and actions, enabling realistic interactions with reduced system complexity.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enables the creation of detailed digital models of local environments, allowing for accurate rendering of real-world objects and realistic interactions, thereby improving communication and reducing computational requirements for 3D communication.
Implementation Method 1
External facing environment cameras positioned on the XR device or in communication with the XR device capture a local physical environment of a user. The external environment cameras make use of depth sensing to create a textured map of a room.
Data Source
AI summary
Disclosed herein is an environmental scanning tool that generates a digital model representing the surroundings of a user of an extended reality head-mounted display device. The environment is imaged in both a depth map and in visible light for some select objects of interest. The selected objects exist within the digital model at higher fidelity and resolution than the remaining portions of the model in order to manage the storage size of the digital model. In some cases, the objects of interest are selected, or their higher fidelity scans are directed, by a remote user. The digital model further includes time stamped updates of the environment such that users can view a state of the environment according to various timestamps.