Recurrent 3D CNN for Real-Time Gesture Classification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems for dynamic hand gesture recognition face challenges in simultaneously detecting and classifying gestures from continuous unprocessed visual data, relying heavily on the nucleus phase for classification and suffering from noise and variability in gesture duration, which limits accuracy and response time.

Innovation Solution

A method using a recurrent three-dimensional convolutional neural network (R3DCNN) that extracts spatio-temporal features from unsegmented data streams, enabling unified detection and classification of hand gestures without preliminary segmentation, and employing connectionist temporal classification (CTC) to predict class labels during the nucleus phase, reducing lag and improving accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional systems use separate detection and classification classifiers processed in sequence, then the system can handle variability in gesture duration and reduce noise, but the accuracy is limited to the accuracy of the upstream gesture detection classifier

Engineering Contradiction:

Improvegesture classification accuracyVSAvoidclassifier architecture complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent combines the detection classifier and recognition classifier into a single unified classifier that performs both gesture detection and classification simultaneously. This unified approach eliminates the accuracy limitation imposed by sequential processing while maintaining the ability to handle gesture duration variability and noise through appropriate loss function design and training procedures.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The unified classifier is designed to perform multiple functions: detecting gesture boundaries, classifying gesture types, and handling variable gesture durations all within a single model. This multi-functional design improves accuracy by allowing the model to learn the relationships between detection and classification tasks jointly, rather than treating them as separate sequential steps.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Measurement precision

If classifiers rely primarily on the nucleus phase for gesture classification, then classification accuracy improves, but the system cannot detect or classify gestures before completion

Engineering Contradiction:

Improvegesture classification accuracyVSAvoidgesture detection latency

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The unified classifier is trained to perform classification during the nucleus phase of gestures, enabling the system to detect and classify gestures before they fully complete. This preliminary action reduces latency by avoiding the need to wait for gesture completion, while maintaining accuracy through appropriate loss functions that account for partial gesture information.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adapts to different gesture phases by using a unified classifier that can operate effectively across preparation, nucleus, and retraction phases. The classifier learns to weight different phases appropriately, focusing on the nucleus phase for accurate classification while still being able to detect gestures in progress during earlier phases.

Inventive Principle:

Principle #15Dynamics

3Productivity

If the system processes continuous streams of unprocessed visual data, then real-time gesture recognition is enabled, but the system must simultaneously detect and classify gestures without preliminary segmentation

Engineering Contradiction:

Improvereal-time processing capabilityVSAvoidgesture boundary detection difficulty

Core Design Contradiction:

ProductivityVSDifficulty of detecting and measuring

Solution Approach 1:

The unified classifier merges the detection and classification functions into a single processing pipeline that operates on continuous video streams without requiring preliminary segmentation. This approach enables real-time processing by eliminating the sequential steps of separate detection and classification, while the model learns to handle gesture boundary detection as part of its overall classification task.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS10157309B2Online detection and classification of dynamic gestures with recurrent convolutional neural networks

Publication Date: 2018.12.18 NVIDIA CORP

AI summary

A method, computer readable medium, and system are disclosed for detecting and classifying hand gestures. The method includes the steps of receiving an unsegmented stream of data associated with a hand gesture, extracting spatio-temporal features from the unsegmented stream by a three-dimensional convolutional neural network (3DCNN), and producing a class label for the hand gesture based on the spatio-temporal features.