HMD Real-World Object Masking via Dynamic Surface Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed-reality computing devices, such as head-mounted displays, struggle to effectively mask out real-world objects while rendering virtual environments, leading to intrusive holographic content and limited user control over visible real-world elements.
Innovation Solution
The HMD device employs a sensor package for surface reconstruction and dynamic masking of real-world objects, allowing users to select which objects remain visible in the virtual reality environment, using techniques like depth sensing and gaze tracking to maintain context as the user moves or objects change position.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If holographic content is rendered over real world objects in mixed-reality environments, then virtual content visibility is improved, but real world object visibility and user control are degraded
Solution Approach 1:
The system dynamically adjusts the masking of real-world objects based on user gaze direction and head pose. The sensor package continuously tracks user movements and updates the masking regions in real-time, allowing the system to adaptively balance virtual content visibility with real-world object visibility according to user attention and context.
2Ease of operation
If surface reconstruction techniques are used to dynamically detect edges and surfaces of real world objects, then real world object visibility is improved, but device complexity increases
Solution Approach 1:
The system extracts only the essential geometric information (edges and surfaces) from the real-world environment using surface reconstruction techniques, rather than processing complete 3D models. This extraction approach maintains real-world object visibility while reducing computational complexity by focusing only on the critical masking boundaries needed for the application.
3Measurement precision
If masked out areas are used to prevent holographic content from overlaying real world objects, then real world object visibility is improved, but virtual environment immersion is degraded
Solution Approach 1:
The system applies masking selectively only to specific regions where real-world objects are detected, rather than applying global masking to the entire field of view. This local approach preserves the precision of real-world object detection while maintaining virtual environment immersion in areas where masking is not required, allowing virtual content to remain visible in unmasked regions.
Data Source
AI summary
An HMD device renders a virtual reality environment in which areas of the real world are masked out so that real world objects such as computer monitors, doors, people, faces, and the like appear visible to the device user and no holographic or virtual reality content is rendered over the visible objects. The HMD device includes a sensor package to support application of surface reconstruction techniques to dynamically detect edges and surfaces of the real world objects and keep objects visible on the display as the user changes position or head pose or when the real world objects move or their positions are changed. The HMD device can expose controls to enable the user to select which real world objects are visible in the virtual reality environment.