Gesture-Based Content Selection Using Fingertip and Depth Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing touch-based and machine vision-based technologies for selecting digital and non-digital content are limited by the requirement for touchscreen hardware, limited gesture recognition, computational complexity, and difficulty in detecting obscured content.
Innovation Solution
A machine vision-based system using two-handed gestures and depth information for accurate content selection, incorporating a deep neural network for hand detection and pose recognition, with optional content recognition in a database to reduce computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple neural networks with many layers are used for gesture detection and content recognition, then detection accuracy is improved, but computational complexity and resource consumption increase
Solution Approach 1:
The patent segments the complex gesture recognition task into distinct components: hand detection module, fingertip location detection module, and gesture recognition module. Each module processes specific aspects independently, reducing the overall computational burden while maintaining accuracy. The hand detection module first identifies hand regions, then fingertip locations are detected within those regions, and finally gestures are recognized based on the spatial relationships between fingertips.
Solution Approach 2:
The patent performs preliminary hand detection and fingertip location detection before gesture recognition. By pre-identifying hand regions and key fingertip positions, the system reduces the computational complexity of the subsequent gesture recognition task. This preliminary processing allows the gesture recognition module to focus only on analyzing the spatial relationships between already-detected fingertips rather than processing raw image data from scratch.
2Adaptability or versatility
If machine vision-based technology is used to detect hand gestures for content selection, then touchscreen hardware requirement is eliminated, but the ability to detect obscured content is reduced
Solution Approach 1:
The patent introduces depth information as an additional dimension to enhance gesture detection. By incorporating depth data from depth sensors, the system can distinguish between hands that are physically touching the content and hands that are merely in front of the content plane. This dimensional addition allows accurate content selection even when hands partially obscure the view, as the depth information reveals the spatial relationship between the hand and the content.
3Device complexity
If single-handed gestures are recognized, then system complexity is reduced, but selection accuracy for regions of interest is limited
Solution Approach 1:
The patent creates a universal gesture recognition system that handles both single-handed and two-handed gestures through the same detection and recognition pipeline. The system detects all hands in the scene, identifies fingertip locations for each hand, and then determines gestures based on the spatial relationships between fingertips regardless of whether one or two hands are present. This multi-functional approach maintains system complexity at a manageable level while significantly improving content selection accuracy for regions of interest.
Data Source
AI summary
Methods and devices for machine vision-based selection of content are described. One or more hands are detected in a current frame of video data. A respective fingertip location is determined for each of up to two of the detected hands. A content selection gesture is determined corresponding to the up to two detected hands. Selected content is extracted, as indicated by the content selection gesture and based on the up to two fingertip locations. The device may be a smartphone, a tablet, a laptop, a smart light device, a reader device, etc.