AR Hand Tracking via Camera and IMU Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) devices face challenges in accurately obtaining three-dimensional position information of hand joints due to limitations in hand detection within the camera's field of view, leading to decreased user satisfaction and service stability.
Innovation Solution
An AR device that uses a combination of camera image frames and inertial measurement unit (IMU) sensor data from an external wearable device to detect hand joints, correct three-dimensional position coordinate values, and update them based on sensor measurement variations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If vision-based hand tracking technology is used to detect the user's hand from camera images, then the freedom of both hands is ensured and no separate external input device is needed, but the accuracy of three-dimensional position coordinate values of hand joints decreases when the hand is not detected or not correctly detected within the camera's field of view
Solution Approach 1:
The patent combines vision-based hand tracking (camera) with inertial measurement unit (IMU) sensor data from an external wearable device to obtain three-dimensional position coordinate values of hand joints. This fusion of multiple sensing modalities ensures continuous and accurate hand tracking even when the hand moves outside the camera's field of view, thereby maintaining both hand freedom and measurement precision.
Solution Approach 2:
The patent introduces an external wearable device equipped with IMU sensors as an intermediary to assist hand tracking. This wearable device acts as a mediator that provides complementary motion data to the vision system, ensuring accurate joint position estimation even when visual detection fails or is unreliable.
2Device complexity
If only camera-based detection is used to obtain three-dimensional position coordinate values of hand joints, then the device complexity is reduced, but the reliability of hand detection and pose/gesture detection decreases
Solution Approach 1:
The patent merges camera-based vision tracking with IMU sensor data from an external wearable device to improve the reliability of hand detection and pose/gesture recognition. This multi-sensor fusion approach compensates for the limitations of each individual sensing modality, providing more robust and stable hand tracking performance.
Solution Approach 2:
The patent changes the sensing parameters by incorporating additional sensor types (IMU sensors measuring acceleration, angular velocity, and magnetic field) alongside the camera system. This expansion of measurement parameters enables more reliable hand detection and tracking under varying conditions.
Data Source
AI summary
Provided are an augmented reality (AR) device for improving the accuracy of position information of a joint of a hand obtained through vision-based hand tracking technology by using a sensor measurement value obtained from an external device, and an operating method thereof. According to an embodiment of the disclosure, the AR device may update a three-dimensional position coordinate value of a joint of a hand by correcting a three-dimensional position coordinate value of the joint of the hand obtained from a plurality of image frames obtained through a camera by using a measurement value of an inertial measurement unit (IMU) sensor obtained from a wearable device.