AR Virtual Keyboard With Hand-Tracked Text Entry
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AR systems lack effective user input modalities for indicating user intent and invoking actions or applications, as they often lack physical input devices like touchscreens or keyboards, making interactions more complicated compared to mobile devices.
Innovation Solution
Combining hand-tracking with Direct Manipulation of Virtual Objects (DMVO) and gesture input modalities to provide an improved text entry user interface, where users manipulate virtual keyboards using hand movements and gestures detected by the AR system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If AR systems use transparent/semi-transparent displays to enable viewing of surrounding environment, then users can see through the display to view the real world, but users lack effective physical input devices for indicating user intent and invoking actions
Solution Approach 1:
The patent replaces traditional mechanical input devices (physical keyboards, touchscreens) with hand-tracking technology that detects hand gestures and movements in 3D space. The system uses cameras and computer vision algorithms to track hand positions and gestures, converting them into digital input signals for interacting with virtual objects and interfaces in the AR environment.
Solution Approach 2:
The patent introduces virtual keyboards and intermediate software layers that mediate between the user's hand gestures and the underlying computer operations. These virtual interfaces appear in the AR display and translate complex hand movements into simple digital commands, making the interaction more intuitive while bridging the gap between physical and digital input.
2Ease of operation
If AR systems implement hand-tracking with Direct Manipulation of Virtual Objects, then users can interact with virtual keyboards using hand movements, but the system complexity increases
Solution Approach 1:
The patent divides the hand-tracking system into separate functional modules: hand detection module, gesture recognition module, virtual object manipulation module, and text input module. Each module handles a specific aspect of the interaction, making the overall system more manageable and easier to optimize. The virtual keyboard itself is segmented into individual key regions that can be independently tracked and activated.
Solution Approach 2:
The patent transitions from 2D touchscreen interaction to 3D spatial hand gesture interaction. The system captures hand movements in three-dimensional space, allowing users to interact with virtual objects at different depths and positions. This adds a spatial dimension to the interaction, enabling more natural and intuitive control while providing richer feedback through depth perception and hand positioning.
Data Source
AI summary
A gesture-based text entry user interface for an Augmented Reality (AR) device is provided. The AR system detects a start text entry gesture made by a user of the AR system, generates a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys, and provides to the user the virtual keyboard user interface. The AR system determines using the one or more cameras, the user's selection of one or more selected virtual keys of the plurality of virtual keys and generates entered text data based on the one or more selected virtual keys. The AR system provides the entered text data to the user using a display of the AR system.