Hand-Projected Virtual Keyboard for Low-Fatigue AR/VR Text Input

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current virtual reality (VR)/augmented reality (AR) text input methods, such as using a virtual keyboard, are inefficient and difficult to operate, leading to user fatigue and soreness due to the need for precise hand movements and prolonged interaction.

Innovation Solution

A text input method and apparatus that projects a virtual keyboard onto a user's hand, allowing input through hand gestures and contact states detected by an electronic device using a neural network model to identify finger positions and interactions, enabling efficient and accurate text entry without requiring constant visual attention.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If a user uses traditional virtual keyboard input methods in VR/AR, then text input can be completed, but the operation becomes inefficient and difficult, causing user fatigue and soreness

Engineering Contradiction:

Improvetext input operationVSAvoidtext input efficiency

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent projects a virtual keyboard onto the user's hand, creating a copy of the keyboard interface on the body surface. This allows the user to interact with the keyboard using natural hand gestures without needing to reach for external controllers or devices, significantly improving ease of operation and reducing physical fatigue

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system uses the user's own hand as the input surface, eliminating the need for separate input devices. The hand serves both as the controller and the keyboard surface, allowing intuitive gesture-based input that improves both ease of operation and input efficiency

Inventive Principle:

Principle #25Self-service

2Measurement precision

If a user uses traditional virtual keyboard input methods, then text can be entered, but the user must excessively stare at the interface, reducing input accuracy

Engineering Contradiction:

Improvekey press positioning accuracyVSAvoidvisual attention time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent uses different colors to represent different key states and feedback types on the virtual keyboard projected on the hand. Visual feedback through color changes allows users to confirm key presses and receive system responses without needing to continuously stare at the interface, improving positioning accuracy while reducing visual fatigue

Inventive Principle:

Principle #32Color changes

Solution Approach 2:

The system provides immediate visual feedback through color changes and graphical updates when keys are pressed or when the system responds to gestures. This feedback mechanism confirms input accuracy in real-time, eliminating the need for prolonged visual verification and improving both precision and efficiency

Inventive Principle:

Principle #23Feedback

3Productivity

If a user performs long-time suspended input in the air, then text input can be completed, but the user feels sore and fatigued

Engineering Contradiction:

Improvetext input continuityVSAvoiduser physical fatigue

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

The system transforms the user's hand into the input surface, eliminating the need to hold hands in the air for extended periods. By using the hand that is already present and naturally positioned, the system enables continuous input without the physical strain of suspended positioning, improving productivity while preventing fatigue

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent moves the keyboard interface from external space to the user's body surface, adding a new dimension to the interaction. This spatial repositioning allows input without requiring the user to maintain awkward suspended postures, enabling continuous input while reducing physical harm

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

4Productivity

If a virtual keyboard is projected on the user's hand, then input efficiency and accuracy improve, but the system complexity increases

Engineering Contradiction:

Improvetext input efficiencyVSAvoidprojection and detection system

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent uses a single camera to perform multiple functions: capturing hand images, projecting the virtual keyboard, and detecting finger gestures. This multi-functional approach improves input efficiency while avoiding the need for separate complex systems for each function, thereby limiting the increase in overall system complexity

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12517646B2Text input method and apparatus based on virtual keyboard

Publication Date: 2026.01.06 HUAWEI TECH CO LTD

AI summary

A text input method and apparatus based on a virtual keyboard are provided, which relate to the field of human-computer interaction, and resolve a problem of how to complete text input based on the virtual keyboard. The method includes an electronic device that detects key points of a first hand based on an image including the first hand after obtaining the image including the first hand, and projects a virtual keyboard on the first hand based on the key points of the first hand. Further, after obtaining an image including the first hand and a finger of a second hand, the electronic device identifies, based on the image including the first hand and the finger of the second hand, a contact state or a pressing state between the finger of the second hand and the first hand.