Depth-Aware Real-World Object Merging in Virtual Reality Headsets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtual reality head-mounted displays (HMDs) obstruct the view of real-world objects, leading to a loss of immersion, realism, and sense of presence, requiring users to remove the HMD to interact with physical equipment, which disrupts training and simulation experiences.
Innovation Solution
A method and apparatus for selectively merging real-world objects into a virtual environment by receiving inputs for virtual and real-world environments, identifying objects within a depth range using depth information, and generating a merged rendering that excludes background objects, allowing real-world objects to be integrated into the virtual environment in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a virtual reality head-mounted display is used to provide immersive virtual environment, then virtual reality immersion and training capability are improved, but the user's ability to see and interact with real-world objects deteriorates
Solution Approach 1:
The patent merges the real-world camera feed with the virtual environment rendering to create a composite display that shows both virtual and real-world objects simultaneously. This is achieved by compositing the camera image with depth information onto the virtual environment, allowing users to access both virtual immersion and real-world visibility through a single HMD display.
2Reliability
If the HMD displays only virtual environment, then virtual reality experience is improved, but the user must remove the HMD to interact with physical equipment, disrupting training continuity
Solution Approach 1:
The patent enables continuous training experience by allowing users to interact with both virtual and real-world objects without removing the HMD. The merged display showing real-world equipment through the camera feed allows users to maintain the training simulation continuity while accessing physical equipment, eliminating the need to take the HMD on and off.
3Adaptability or versatility
If real-world objects are merged into virtual environment, then immersion and realism are improved, but processing complexity and computational requirements increase
Solution Approach 1:
The patent extracts only the necessary portions of the real-world environment (objects within depth range and differentiable from background) and merges them with the virtual environment. This selective extraction approach reduces processing complexity by focusing only on relevant real-world objects rather than processing the entire camera feed, while still achieving improved immersion and realism.
Data Source
AI summary
Methods and systems for selectively merging real-world objects into a virtual environment are disclosed. The method may include: receiving a first input for rendering of a virtual environment, a second input for rendering of a real-world environment, and a depth information regarding the rendering of the real-world environment; identifying at least one portion of the rendering of the real-world environment that is within a depth range and differentiable from a predetermined background; generating a merged rendering including the at least one portion of the rendering of the real-world environment into the rendering of the virtual environment; and displaying the merged rendering to a user.