3D Hand Tracking for Overlapping Interactions in Wearable Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object detection pipelines struggle with efficiently detecting and tracking multiple instances of interacting hands, particularly when they overlap in 2D space, due to the reliance on non-maximum suppression methods that filter out overlapping detections.
Innovation Solution
A method utilizing a wearable multimedia device with a depth sensor to obtain 2D and 3D data, detect hand regions, landmarks, and track hands in 3D space by determining confidence scores and world coordinates, avoiding the need for non-maximum suppression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If non-maximum suppression (NMS) is used to filter proposals, then processing efficiency is improved, but detection accuracy for overlapping hands deteriorates
Solution Approach 1:
The patent transitions from 2D image space to 3D depth space by utilizing depth information from the sensor. By detecting hands in three-dimensional space rather than two-dimensional image space, the system can distinguish overlapping hands that appear merged in 2D but are separated in 3D, thereby maintaining detection accuracy while avoiding the need for NMS filtering.
2Quantity of substance
If multiple hand instances are detected in 2D space, then detection completeness is improved, but false positives increase due to overlapping detections
Solution Approach 1:
The system uses depth information to differentiate between multiple hand instances that overlap in 2D space. By projecting hands into 3D space and using depth coordinates, the system can reliably distinguish and track multiple hands without generating false positives from overlapping detections.
Solution Approach 2:
The patent introduces depth information as an intermediary dimension between 2D image data and hand detection. This intermediate 3D representation serves as a mediator that resolves the ambiguity of overlapping 2D hand detections, allowing the system to accurately identify and track multiple hands simultaneously.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables simultaneous and accurate detection of multiple interacting hands, even when they are close together, by leveraging 3D tracking and landmark detection models.
Implementation Method 1
the sensor is a time of flight camera that outputs infrared or amplitude image data and the depth data that are registered by the sensor
Data Source
AI summary
Systems, methods, devices and non-transitory, computer-readable storage mediums are disclosed for the tracking of interacting hands using a sensor of a wearable multimedia device. In an embodiment, a method comprises: obtaining, with a sensor of a wearable multimedia device, a first frame of two-dimensional (2D) image data and a second frame of three-dimensional (3D) depth data; determining multiple hand regions in the first frame of 2D image data; determining a location of each hand region in the first frame of 2D image data; detecting at least one landmark in each detected hand region; generating a confidence score for each landmark detected in each detected hand region; determining 3D world coordinates for each landmark that is visible in the first frame of the 2D image data; tracking each landmark in 3D world coordinates; and determining an interaction with the hands based on the tracking of each landmark in 3D world coordinates.