Scalable Real-Time Hand Tracking With Palm And Landmark Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hand tracking systems are computationally expensive and not suitable for mobile applications due to their reliance on complex computer vision tasks that require powerful desktop environments.
Innovation Solution
A machine-learned palm detection model and hand landmark model are used to detect palms and generate oriented bounding boxes, followed by key-point localization to determine three-dimensional coordinates of hand landmarks, reducing computational complexity and enabling efficient hand tracking on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex computer vision tasks are used for hand tracking, then measurement precision is improved, but use of energy increases and device complexity increases
Solution Approach 1:
The hand tracking system is segmented into two specialized models: a palm detection model that identifies palm regions and generates bounding boxes, and a hand landmark model that detects specific hand keypoints. This segmentation allows each model to be optimized for its specific task, reducing overall computational energy consumption while maintaining high measurement precision for hand tracking
2Measurement precision
If complex computer vision tasks are used for hand tracking, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The system divides the complex hand tracking problem into two simpler, specialized sub-tasks handled by separate models: palm detection and hand landmark detection. This segmentation reduces device complexity by making each individual model simpler and more efficient, while the combination maintains high overall measurement precision
Solution Approach 2:
The palm detection model serves as an intermediary that processes the input image and generates bounding box predictions, which then serve as input to the hand landmark model. This intermediary structure simplifies the overall system by breaking down the complex task into manageable stages, reducing device complexity while preserving measurement precision
3Measurement precision
If powerful desktop environments are used, then measurement precision is improved, but portability decreases
Solution Approach 1:
The patent employs lightweight machine learning models that can be deployed on mobile devices with limited computational resources. These models are optimized to run efficiently on mobile hardware, replacing the need for powerful desktop environments while maintaining acceptable measurement precision for hand tracking applications
Data Source
AI summary
Example aspects of the present disclosure are directed to computing systems and methods for hand tracking using a machine-learned system for palm detection and key-point localization of hand landmarks. In particular, example aspects of the present disclosure are directed to a multi-model hand tracking system that performs both palm detection and hand landmark detection. Given a sequence of image frames, for example, the hand tracking system can detect one or more palms depicted in each image frame. For each palm detected within an image frame, the machine-learned system can determine a plurality of hand landmark positions of a hand associated with the palm. The system can perform key-point localization to determine precise three-dimensional coordinates for the hand landmark positions. In this manner, the machine-learned system can accurately track a hand depicted in the sequence of images using the precise three-dimensional coordinates for the hand landmark positions.