Augmented Reality Hand Interaction via Depth-Image Calibration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems fail to provide intuitive interaction between users and 3D virtual objects, as they lack accurate 3D position recognition and occlusion effects, leading to a loss of immersion when using physical interaction devices.
Innovation Solution
An augmented reality-based hand interaction apparatus and method that extracts 3D mesh information from depth images and calibrates it with RGB camera data to enable intuitive interaction and occlusion rendering, using modules for object recognition, 3D posture estimation, and collision detection to enhance immersion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If RGB image-based hand recognition is used, then the system can identify hand regions, but misrecognition occurs when objects with skin-like colors are present and accurate 3D position information cannot be obtained
Solution Approach 1:
The patent combines RGB camera data with depth camera data to achieve accurate hand recognition and 3D position estimation. The RGB image provides color information for hand segmentation while the depth image provides 3D spatial information, merging both data sources to overcome the limitations of using either alone.
Solution Approach 2:
The patent introduces a depth camera as an intermediary device to capture 3D spatial information. This intermediary sensor bridges the gap between 2D RGB imaging and 3D position recognition, enabling accurate hand tracking without misrecognition issues.
2Adaptability or versatility
If 2D UI interfaces are used in augmented reality, then existing interfaces can be utilized, but the sense of immersion is lost as interaction does not occur in augmented three dimensions
Solution Approach 1:
The patent transitions from 2D UI interfaces to 3D spatial interaction by utilizing depth information. Users can interact with augmented reality content using hand gestures in three-dimensional space, maintaining compatibility with existing interaction paradigms while adding the immersive dimension of spatial awareness.
3Reliability
If virtual content is synchronized with real space in augmented reality, then the sense of immersion is provided, but occlusion effects between augmented 3D content and real objects cannot be produced
Solution Approach 1:
The patent replaces traditional 2D rendering mechanisms with 3D depth-aware rendering. By utilizing depth camera data, the system determines spatial relationships between real objects and virtual content, enabling realistic occlusion effects where virtual objects are correctly hidden behind or visible in front of real objects based on their 3D positions.
Data Source
AI summary
An augmented reality-based hand interaction apparatus and method using image information. The augmented reality-based hand interaction apparatus using image information includes an object recognition module configured to extract feature points from an input color image, recognize an object which is a target of interest based on the extracted feature points, and estimate a three-dimensional (3D) posture of the recognized object, and an interaction module configured to extract 3D mesh information from an input depth image, match 3D coordinates of the extracted 3D mesh information with 3D coordinates of the object recognized by the object recognition module to extract 3D position information of the recognized object in 3D space, and detect whether or not an augmented 3D object and the recognized object collide with each other based on relative positional relationship information between the recognized object and the augmented 3D object.