Freehand Typing in AR via Skin-Projected Virtual Keyboard
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text input methods in virtual and augmented reality (VR/AR) are inefficient and require users to adapt unfamiliar motor skills, limiting mobility and causing user fatigue, as they often involve handheld devices or mid-air typing without haptic feedback.
Innovation Solution
A computer-implemented method for freehand typing that tracks hand and thumb movements to display a virtual keyboard on the user's skin surface, allowing thumb typing with haptic feedback, enabling efficient text entry by leveraging familiar smartphone typing skills without the need for handheld devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If handheld devices or mid-air typing are used for text input in VR/AR, then text entry functionality is achieved, but user mobility is limited and typing efficiency is reduced
Solution Approach 1:
The system uses the user's own body (hand skin surface) as the keyboard interface, eliminating the need for external handheld devices. The skin surface itself serves as the typing surface, providing haptic feedback naturally without requiring the user to hold or interact with separate input devices, thus maintaining mobility while enabling efficient typing.
Solution Approach 2:
The patent creates a virtual keyboard that is projected onto or near the user's hand, replicating the familiar smartphone keyboard layout and typing experience. This virtual copy of a physical keyboard allows users to type with familiar motor skills while maintaining the benefits of VR/AR immersion and device-free operation.
2Productivity
If virtual keyboards are displayed in mid-air without haptic feedback, then text input functionality is provided, but typing accuracy decreases and user fatigue increases
Solution Approach 1:
The patent introduces an optical intermediary (virtual keyboard projection) that bridges the gap between the user's thumb and the invisible mid-air interface. By projecting visual keyboard boundaries and key locations onto or near the hand, the system provides a visual mediator that guides thumb placement and typing actions, improving accuracy without requiring physical contact with a traditional keyboard.
3Adaptability or versatility
If unfamiliar typing methods are implemented in VR/AR, then text input capability is achieved, but learning curve increases and user adaptation becomes difficult
Solution Approach 1:
Instead of requiring users to adapt to new VR/AR-specific typing methods, the patent inverts the approach by bringing the familiar smartphone typing interface into the VR/AR environment. The virtual keyboard replicates the well-known smartphone layout and typing mechanics, allowing users to leverage existing motor skills and muscle memory rather than learning entirely new input methods.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Various aspects of the subject technology relate to systems, methods, and machine-readable media for freehand typing in AR/VR applications. Various aspects may include receiving data from one or more sensors at a client device. Aspects may also include tracking hand position of a user of the client device based on the data. Aspects may also include identifying a performance of a preset posture by the user. Aspects may also include determining a position of a virtual keyboard based on the hand position, wherein the position is determined based on the preset posture being performed and displaying the virtual keyboard at the position. The position may be optimized such that the display of the virtual keyboard enables haptic feedback when the user performs input selections via the virtual keyboard (using at least one hand). The virtual keyboard may be displayed in a general view of the user.