Infrared Camera Hand Pose Estimation via Domain Transfer Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for 3D hand pose estimation, such as generative and discriminative approaches, fail to accurately estimate hand pose during fast hand motions due to motion blur, which contaminates RGB and depth images, and existing solutions like sensor-fusion methods are limited in handling general motion blur and require external sensors.
Innovation Solution
The method employs domain transfer learning using paired infrared (IR) and depth images from an IR camera, where a hand image generator synthesizes a depth image from an IR image, and two hand pose estimators calculate the 3D hand pose, minimizing the impact of motion blur by leveraging the relative robustness of IR images to fast hand motion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If RGB or depth images are used for hand pose estimation, then the method can process hand motion, but motion blur contaminates the images and reduces estimation accuracy during fast hand motion
Solution Approach 1:
The patent introduces an intermediary domain transfer learning approach that maps features from the IR image domain to the depth image domain. The domain adapter layer serves as a mediator that translates IR image features into meaningful depth space representations, enabling accurate hand pose estimation without directly relying on blurred depth images. This intermediary transformation resolves the contradiction by providing a clean feature representation path that avoids motion blur contamination.
Solution Approach 2:
The patent changes the input domain parameter from traditional RGB/depth images to infrared images. By transforming the input modality and using domain transfer learning to map IR features to depth space, the system achieves robust hand pose estimation during fast motion. This parameter change in the input domain fundamentally alters how motion blur affects the system, as IR images are less susceptible to motion blur compared to visible light images.
2Stability of the object's composition
If generative approach is used to track hand articulation, then temporal continuity can be utilized, but fast hand motion causes the approach to fail due to motion blur and inability to handle rapid changes
Solution Approach 1:
The patent performs preliminary action by pre-training the domain adapter layer on paired IR and depth images from slow-motion datasets. This pre-training establishes robust feature mappings before the system encounters fast hand motion. The preliminary learning of domain-invariant features enables the system to handle rapid motion changes without failing, as the adapter is already optimized to transfer features accurately across different motion speeds.
3Adaptability or versatility
If discriminative approach is used to estimate hand pose in single frame, then temporal continuity assumption is not required, but motion blur still affects estimation accuracy when training data does not include blurry images
Solution Approach 1:
The patent creates a copying mechanism through domain transfer learning, where the domain adapter layer learns to replicate and transform IR image features into depth space representations. This copying process enables the system to generate clean depth-like feature maps from IR images, effectively creating a virtual copy of the depth image that is free from motion blur contamination. The copied features then serve as the basis for accurate hand pose estimation.
4Measurement precision
If deblurring method is applied to depth images, then motion blur can be reduced, but existing methods are limited to specific multi-tap ToF cameras and cannot handle general motion blur cases
Solution Approach 1:
The patent replaces the mechanical/deep learning-based deblurring approach with a domain transfer learning approach that operates in the feature space. Instead of trying to reverse the blur convolution operation or process individual camera taps, the system substitutes a neural network-based feature mapping approach that directly transforms IR image features to depth space. This substitution eliminates the need for complex deblurring algorithms and makes the system applicable to any camera type without requiring specific multi-tap ToF capabilities.
Data Source
AI summary
The present invention relates to a hand position estimation method and system for estimating a 3-dimensional hand position for quick hand movement, on the basis of domain transfer learning from a depth image to an infrared image, the hand position estimation method comprising the steps of: processing a depth image and an infrared image for hand movement; synthesizing a depth map with the infrared image by using a hand image generator (HIG), and estimating skeletal positions of hand joints from each of the depth map and an infrared map; and calculating a 3-dimensional hand position by using the skeletal positions and the center of a hand depth image.