Transfer Learning for Hand Gesture Recognition in VR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current virtual reality systems face limitations in user immersion due to the difference between real-world and virtual-world actions, and the process of acquiring and training deep learning-based user hand gesture recognizers is time-consuming and costly.
Innovation Solution
A system utilizing transfer learning to generate and manage a deep learning-based user hand gesture recognizer, incorporating a head-mounted display, hand gesture sensors, and a controller device to acquire and process hand data, convert it into gesture data, and execute content based on recognition results, allowing for the application of existing recognizers across different data acquisition methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning-based user hand gesture recognizer is trained from scratch using acquired user hand data, then recognition accuracy is improved, but training time and cost increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training a deep learning model on a large-scale hand gesture dataset before fine-tuning it with user-specific data. This allows the system to start with a model that already has general gesture recognition capabilities, significantly reducing the training time required while maintaining high accuracy through subsequent fine-tuning with smaller amounts of user data.
Solution Approach 2:
The patent utilizes parameter changes by adjusting model parameters during transfer learning from general gesture recognition to user-specific gesture recognition. The system modifies weights and biases of pre-trained model parameters based on user feedback and specific user hand data, enabling adaptation to individual users without requiring complete retraining from scratch.
2Measurement precision
If deep learning-based user hand gesture recognizer is trained from scratch using acquired user hand data, then recognition accuracy is improved, but training cost increases significantly
Solution Approach 1:
The patent applies copying by replicating and adapting pre-trained model architectures and weight initializations for different users and applications. Instead of creating new models from scratch for each user, the system copies the proven architecture and pre-trained weights, then fine-tunes them with minimal user-specific data, significantly reducing computational costs and resource requirements.
Solution Approach 2:
The patent utilizes parameter changes by adjusting model parameters during transfer learning from general gesture recognition to user-specific gesture recognition. The system modifies weights and biases of pre-trained model parameters based on user feedback and specific user hand data, enabling adaptation to individual users without requiring complete retraining from scratch.
3Adaptability or versatility
If multiple hand gesture recognizers are trained for different data acquisition methods, then adaptability to various sensors is improved, but system complexity increases
Solution Approach 1:
The patent applies universality by designing a single deep learning model architecture that can process multiple types of hand gesture data from different sensors (cameras, data gloves, thimbles). The model is trained to be sensor-agnostic, accepting various input formats and normalizing them into a unified representation, thereby eliminating the need for separate recognizers for each sensor type while maintaining high adaptability.
Solution Approach 2:
The patent uses an intermediary approach by introducing a data normalization and feature extraction layer that mediates between different sensor inputs and the core recognition model. This intermediary layer converts diverse sensor data formats into a standardized intermediate representation, allowing the main recognizer to process all inputs uniformly without requiring multiple specialized models.
Data Source
AI summary
Provided is a system for recognizing a user hand gesture and providing virtual reality content based on deep learning using transfer learning. The system includes a head mounted display (HMD) configured to display, to a user, virtual reality content based on a hand gesture, a hand gesture sensor configured to acquire hand data of the user, and a controller device configured to receive the hand data and convert the received hand data into hand gesture data, manage a hand gesture recognizer on the basis of deep learning, and perform content execution management.