Gesture-Based Virtual Keyboard for Focused AR Text Entry
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AR systems lack effective user input modalities for indicating user intent and invoking actions due to the absence of physical input devices like touchscreens or keyboards, making interactions more complicated compared to mobile devices.
Innovation Solution
Combining hand-tracking with Direct Manipulation of Virtual Objects (DMVO) and gesture recognition to provide a user interface with a virtual keyboard that allows users to input text by manipulating virtual keys with one hand and making gestures with the other, using continuous motion or an infinite ray cursor for text entry.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If physical input devices like touchscreens or keyboards are used, then text entry efficiency is improved, but device portability and AR immersion are compromised
Solution Approach 1:
The patent replaces physical mechanical input devices (keyboards, touchscreens) with a gesture-based interaction system. The system uses camera tracking to detect hand gestures and translates them into text input commands, eliminating the need for physical input hardware while maintaining text entry functionality in AR environments
Solution Approach 2:
The patent introduces an intermediary software layer that captures raw gesture data from camera tracking, processes the gestures through recognition algorithms, and translates them into meaningful text input commands. This intermediary processing layer enables complex text entry functionality without requiring physical input devices
2Ease of operation
If traditional button presses or touchpad interactions are used, then text entry is achievable, but user focus must be diverted from the real-world view
Solution Approach 1:
The patent moves the input interface from a 2D screen-based interaction (requiring user gaze diversion) to 3D spatial gesture recognition in the real-world environment. Users can perform text entry gestures naturally in space while maintaining their visual focus on the real-world scene through the AR display
Solution Approach 2:
The system enables users to perform text entry using their own natural hand movements and gestures without requiring them to shift attention to a separate input interface. The gesture recognition system continuously monitors and interprets hand movements in the user's natural field of view, allowing simultaneous real-world observation and text input
3Adaptability or versatility
If gesture recognition is implemented, then input versatility is improved, but detection precision and reliability are reduced
Solution Approach 1:
The patent segments the gesture recognition process into multiple distinct stages: raw camera data capture, hand landmark detection, gesture pattern recognition, and text input translation. Each stage processes specific aspects of the input signal with specialized algorithms, improving overall detection precision while maintaining gesture versatility
Solution Approach 2:
The system implements feedback mechanisms where recognized gestures provide immediate visual confirmation to the user, and continuous tracking data refines gesture recognition over time. This feedback loop enhances detection reliability by allowing the system to adjust and confirm gesture interpretations based on ongoing motion patterns
Data Source
AI summary
A gesture-based text entry user interface for an Augmented Reality (AR) system is provided. The AR system detects a start text entry gesture made by a user of the AR system, generates a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys, and provides to the user the virtual keyboard user interface. The AR system detects a hold of an enter text gesture made by the user. While the user holds the enter text gesture, the AR system collects continuous motion gesture data of a continuous motion as the user makes the continuous motion through the virtual keys of the virtual keyboard. The AR system detects a release of the enter text gesture by the user and generates entered text data based on the continuous motion gesture data.