Mixed Reality Display Using 3D Surface Reconstruction Mesh and Live Video Overlay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video feeds in medical and other settings are limited as they do not allow remote observers to alter the field of view or observe contextual information outside the captured area, providing a static and two-dimensional experience.
Innovation Solution
A computerized method and system that combines 3D surface reconstruction mesh data with live video data to create a mixed reality environment, enabling remote users to observe and interact with a local environment through a remote mixed reality device, using photogrammetry to generate a 3D space that matches the local environment and overlaying live video feeds for a fully immersive experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If static video feeds are used for remote observation, then the system is simple and easy to implement, but the field of view cannot be altered and contextual information outside the captured area cannot be observed
Solution Approach 1:
The patent combines 3D surface reconstruction mesh data with live video feeds to create a mixed reality environment. The 3D mesh provides contextual information about the entire environment while the live video feed shows the current field of view, allowing remote users to observe both the captured area and surrounding context simultaneously.
Solution Approach 2:
The patent transitions from 2D static video feeds to a 3D mixed reality environment by generating 3D surface reconstruction meshes from optical data. This dimensional enhancement allows remote users to navigate and observe the environment from multiple perspectives, altering the field of view dynamically while maintaining system feasibility.
2Loss of information
If 3D surface reconstruction mesh and live video overlay are combined, then immersive and interactive observation is enabled, but data processing and transmission requirements increase
Solution Approach 1:
The patent segments the visual information into two distinct components: 3D surface reconstruction mesh data providing structural and contextual information, and live video feeds providing real-time visual details. This segmentation allows efficient processing and transmission of different data types with optimized bandwidth requirements.
Solution Approach 2:
The patent uses 3D surface reconstruction meshes as an intermediary that bridges the gap between static environmental context and dynamic live video feeds. The mesh acts as a persistent spatial framework that enables navigation and context awareness without requiring continuous transmission of full environmental data.
3Measurement precision
If photogrammetry is used to generate 3D space, then the virtual environment matches the local environment accurately, but the processing time and computational resources increase
Solution Approach 1:
The patent performs photogrammetry processing in advance to generate the 3D surface reconstruction mesh before the remote observation session begins. This preliminary action creates a ready-to-use spatial framework that can be quickly loaded and displayed, avoiding real-time processing delays during the actual observation and interaction phase.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This solution enhances the remote user experience by providing a hybrid virtual reality experience that allows for immersive and interactive observation of the local environment, enabling more effective guidance and feedback in medical settings and other applications.
Implementation Method 1
generate a 3D surface reconstruction mesh of the surface from the obtained optical data using photogrammetry
Data Source
AI summary
The disclosure herein describes enabling a user of a remote mixed reality (MR) device to observe an environment of a local MR device combined with 3D surface reconstruction (SR) mesh data and live video data. Optical data of a surface of an environment is obtained and a 3D surface reconstruction mesh of the surface is generated from the obtained optical data using photogrammetry. The generated 3D surface reconstruction mesh is provided for display by a remote device. A live video feed of a window region of the environment is obtained and the live video feed of the window region is provided for display on the generated 3D surface reconstruction mesh by the remote device. Further, a remote user is enabled to provide feedback to a user of the local MR device, including audio feedback such as speech and virtual artifacts that are displayed to the local user.