Mobile Hand Tracking Using Palm Detection and 3D Landmarks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hand tracking and perception systems are computationally expensive and not suitable for mobile applications due to their reliance on complex and powerful desktop environments, making them inefficient for real-time hand detection and tracking.
Innovation Solution
A machine-learned palm detection model and hand landmark model are used to detect palms and generate oriented bounding boxes, followed by key-point localization to determine three-dimensional coordinates of hand landmarks, reducing computational complexity and enabling efficient hand tracking on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex hand tracking systems are used to achieve accurate hand detection and tracking, then measurement precision is improved, but device complexity and computational requirements increase making them unsuitable for mobile applications
Solution Approach 1:
The hand tracking system is divided into two separate machine-learned models: a palm detection model that identifies palm locations and generates bounding boxes, and a hand landmark model that detects specific hand keypoints within those bounding boxes. This segmentation allows each model to be optimized independently for mobile deployment while maintaining overall tracking accuracy.
Solution Approach 2:
The system extracts only the necessary region of interest (palm area) from the full image frame using bounding boxes generated by the palm detection model. This extraction reduces the computational burden on the hand landmark model by focusing processing only on relevant regions, enabling efficient real-time tracking on mobile devices.
2Productivity
If complex hand tracking systems are deployed to achieve real-time tracking capability, then productivity is improved, but use of energy and computational resources increase making them unsuitable for mobile devices
Solution Approach 1:
By segmenting the tracking task into palm detection and hand landmark detection phases, the system processes only relevant image regions in the second phase. This reduces overall computational energy consumption while maintaining real-time tracking productivity on mobile devices.
Solution Approach 2:
The palm detection model performs preliminary action by identifying and bounding palm regions before the hand landmark model processes the image. This preliminary filtering reduces the computational workload for subsequent detailed hand analysis, enabling real-time tracking with lower energy consumption.
3Measurement precision
If full image frames are processed for hand landmark detection, then measurement precision is improved, but loss of time increases due to processing large image areas
Solution Approach 1:
The system extracts and processes only the bounding box regions containing palms rather than full image frames. This extraction maintains hand landmark detection precision by focusing on relevant areas while significantly reducing processing time and computational resources required.
Solution Approach 2:
The hand landmark model applies local quality processing by focusing computational resources only on regions identified as containing palms. This localized processing approach maintains detection accuracy in critical areas while reducing overall processing time compared to analyzing entire image frames.
Data Source
AI summary
Example aspects of the present disclosure are directed to computing systems and methods for hand tracking using a machine-learned system for palm detection and key-point localization of hand landmarks. In particular, example aspects of the present disclosure are directed to a multi-model hand tracking system that performs both palm detection and hand landmark detection. Given a sequence of image frames, for example, the hand tracking system can detect one or more palms depicted in each image frame. For each palm detected within an image frame, the machine-learned system can determine a plurality of hand landmark positions of a hand associated with the palm. The system can perform key-point localization to determine precise three-dimensional coordinates for the hand landmark positions. In this manner, the machine-learned system can accurately track a hand depicted in the sequence of images using the precise three-dimensional coordinates for the hand landmark positions.