AR Hand Point-Cloud Rendering for Accurate Virtual Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR gesture interaction methods fail to accurately render the occlusion relationship between hands and virtual objects, leading to unrealistic user interactions in augmented reality scenes.
Innovation Solution
Utilizing a combination of RGB images, depth images, and inertial measurement unit data to perform three-dimensional dense modeling, enabling accurate rendering of the occlusion relationship between hands and virtual objects through three-dimensional point cloud information and pose information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If 2D SLAM model is used for gesture interaction, then device complexity is reduced, but occlusion relationship rendering between hands and virtual objects cannot be achieved
Solution Approach 1:
The patent transitions from 2D SLAM model to 3D dense modeling by introducing depth dimension through RGB-D images and point cloud generation. This dimensional expansion enables accurate representation of hand geometry and occlusion relationships with virtual objects, resolving the limitation of 2D approaches而无法实现遮挡关系的问题
Solution Approach 2:
The patent combines multiple data sources (RGB images, depth images, IMU data) to create a composite 3D representation of hands through point cloud generation. This composite approach integrates visual and inertial information to achieve accurate occlusion rendering while maintaining system feasibility
2Manufacturing precision
If 3D dense modeling with RGB-D images is implemented, then occlusion relationship accuracy is improved, but use of energy increases
Solution Approach 1:
The patent segments the hand modeling process into distinct stages: RGB-D image acquisition, point cloud generation, pose estimation, and occlusion rendering. This segmentation allows selective processing and optimization of energy-intensive operations, reducing overall energy consumption while maintaining accuracy
Solution Approach 2:
The patent implements selective 3D dense modeling only for hand regions identified in the scene, rather than processing entire frames. This partial action approach reduces computational load and energy consumption while maintaining accurate occlusion rendering where needed
Data Source
Figure 1
Figure 2~3
Figure 4~6
AI summary
Disclosed area an AR scenario-based gesture interaction method, an AR scenario-based gesture interaction apparatus, a computer-readable medium, and a communication terminal, relating to the field of augmented reality technology. Said method comprises: acquiring an RGB image and a depth image of a current frame, and corresponding IMU data (S11); processing the RGB image, the depth image and the IMU data so as to acquire attitude information and hand information of the current frame (S12); performing three-dimensional dense modeling according to the attitude information and the hand information of the current frame so as to acquire three-dimensional point cloud information of the hand in a reference coordinate system (S13); acquiring pose information of a virtual object in the reference coordinate system (S14); and on the basis of the three-dimensional point cloud information of the hand and the pose information of the virtual object in the reference coordinate system, and preset point cloud information of the virtual object, rendering the hand and the virtual object, so as to acquire a shielding relationship between the hand and the virtual object (S15).