Deep Learning Hand Gesture Recognition in AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional wearable Augmented Reality (AR) devices struggle with hand gesture recognition in varying light conditions and reflective surfaces, requiring complex and expensive hardware, and current methods are limited by skin color dependency and lack robustness in realistic conditions.
Innovation Solution
A deep learning-based system for detecting and classifying dynamic hand gestures using a single RGB camera without depth sensors, employing a multi-layer LSTM network to estimate temporal information and classify gestures, overcoming limitations of existing techniques by utilizing RGB image data for robust gesture recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional wearable AR devices use complex on-board sensors and customized processing chips for hand gesture recognition, then gesture recognition accuracy is improved, but device cost and complexity increase
Solution Approach 1:
The patent extracts the depth sensing functionality from the AR device hardware and performs gesture recognition using only the standard RGB camera already present in frugal devices. By taking out the requirement for specialized depth sensors and processing chips, the system achieves gesture recognition without increasing device complexity or cost.
Solution Approach 2:
The system creates a virtual depth map by processing RGB image data through deep learning algorithms. This copied representation of depth information allows the system to achieve accurate gesture recognition without physical depth sensors, resolving the contradiction between accuracy and hardware complexity.
2Reliability
If conventional AR devices use specialized depth sensors for gesture detection, then gesture recognition reliability is improved, but device cost increases
Solution Approach 1:
The patent replaces expensive, specialized depth sensors with a standard RGB camera that is already present in frugal AR devices. This substitution uses cheaper, widely available hardware to achieve reliable gesture recognition, eliminating the need for costly specialized components.
Solution Approach 2:
The system changes the input parameter from depth information (requiring specialized sensors) to RGB color information (available from standard cameras). By processing RGB data through multi-layer LSTM networks, the system achieves reliable gesture recognition using parameters already captured by inexpensive hardware.
3Ease of operation
If conventional gesture recognition methods use skin color detection, then detection simplicity is improved, but robustness in varying light conditions deteriorates
Solution Approach 1:
The patent introduces a deep learning-based hand pose estimation system as an intermediary between the RGB camera and gesture recognition. This intermediary extracts hand keypoint coordinates through multi-layer LSTM networks, providing a robust representation that is independent of skin color and lighting conditions, thereby resolving the contradiction between simplicity and robustness.
4Measurement precision
If conventional systems use multiple cameras for 3D hand pose detection, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent makes the single RGB camera perform multiple functions: capturing images for display, extracting hand keypoints for gesture recognition, and inferring 3D hand pose information. This multi-functionality allows the system to achieve 3D pose detection accuracy without adding multiple cameras, resolving the contradiction between precision and complexity.
Solution Approach 2:
The system creates virtual 3D hand pose representations by processing 2D RGB image data through deep learning algorithms. This copied 3D information from 2D images enables accurate hand pose detection without physical multi-camera systems, eliminating the need for complex hardware while maintaining measurement precision.
Data Source
Figure 1A~1D
Figure 2~3
Figure 4
AI summary
A system and method for hand-gesture recognition are provided. The method includes receiving frames of a media stream of a scene captured from a first person view (FPV) of a user using at least one RGB sensor communicably coupled to a wearable AR device. The media stream includes RGB image data associated with the frames of the scene. The scene comprises a dynamic hand gesture performed by the user. A temporal information associated with the dynamic hand gesture is estimated from the RGB image data by using a deep learning model. The estimated temporal information is associated with hand poses of the user and comprising a plurality of key-points identified on user's hand in the plurality of frames. Based on the temporal information of the key points, the dynamic hand gesture is classified into at least one predefined gesture class by using a multi-layered LSTM classification network.