3D Hand Occlusion Rendering for Head-Mounted Virtual Screens
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
When a realistic scene includes a user's hand, virtual objects can occlude the hand, leading to a poor image display effect in head-mounted devices.
Innovation Solution
Determine if a hand object overlaps with a virtual screen, select target mesh information to represent three-dimensional spatial information, crop a target image from the realistic scene, and combine it with mesh information to create a three-dimensional hand image, then display the virtual screen and hand image on a head-mounted device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the virtual object is superimposed in front of the realistic scene to ensure complete display of the virtual object, then the virtual object can be fully displayed, but the user's hand may be occluded by the virtual object
Solution Approach 1:
The patent segments the realistic scene by detecting and extracting the hand object from the background scene. The hand is identified as a separate entity with its own spatial coordinates, allowing it to be processed independently from the rest of the scene. This segmentation enables the system to apply different rendering rules to the hand versus other scene elements.
Solution Approach 2:
The patent transitions from two-dimensional image processing to three-dimensional spatial reasoning by obtaining depth information for the hand object and virtual objects. By using 3D coordinates and depth values, the system can determine spatial relationships and occlusion relationships in three-dimensional space, allowing the hand to correctly occlude virtual objects when the hand is closer to the user.
2Stability of the object's composition
If the virtual object is placed at a preset spatial position to maintain stable display, then the display position is consistent, but it may overlap with and occlude the hand object
Solution Approach 1:
The patent makes the display composition dynamic by continuously tracking the hand's position and adjusting the rendering accordingly. While virtual objects maintain their preset spatial positions for stability, the system dynamically determines occlusion relationships based on the hand's real-time 3D position, allowing the hand to occlude virtual objects when appropriate without requiring the virtual objects to move.
Data Source
AI summary
The embodiments of the present disclosure provide an image display method, an image display apparatus, a storage medium, and an electronic device. The method includes: determining, in response to receiving a realistic scene image including a hand object, whether the hand object overlaps with a virtual screen at a preset spatial position; selecting target mesh information corresponding to the hand object from three-dimensional mesh information corresponding to the realistic scene image when the hand object overlaps with the virtual screen; cropping a target image corresponding to the hand object from the realistic scene image, and combining the target image with the target mesh information to obtain a three-dimensional hand image corresponding to the hand object; and displaying the virtual screen and the three-dimensional hand image on a display interface of a head-mounted device, based on the preset spatial position and the three-dimensional hand image.