Virtual Interface Interaction via Gesture Recognition

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional virtual reality systems lack a specification for virtual keyboard interaction using gesture recognition, relying on remote or joystick-based humanized user interfaces for interaction.

Innovation Solution

A method and apparatus for virtual user interface interaction based on gesture recognition, which detects two hands, recognizes hand gestures, projects a virtual interface on an open hand, tracks the index fingertip, and interprets its movement as a click command to generate image data for a virtual keyboard, allowing interaction without physical contact.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional humanized user interfaces are used with remote or joystick, then interaction is possible, but user immersion and willingness of VR experience are reduced

Engineering Contradiction:

Improveuser immersionVSAvoidphysical input devices

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent extracts the interaction function from physical devices (remote control, joystick) and relocates it to the user's hands through gesture recognition. The hands themselves become the input interface, eliminating the need for separate physical input devices and enhancing immersion by making the user's own body parts the interaction medium.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent replaces the mechanical system of physical input devices with an optical recognition system. Instead of mechanical buttons and joysticks, the system uses cameras and computer vision algorithms to detect and interpret hand gestures, substituting mechanical interaction with optical-field-based gesture recognition.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If gesture recognition is implemented without specification, then virtual interface interaction is possible, but interaction precision and reliability are insufficient

Engineering Contradiction:

Improvegesture interaction capabilityVSAvoidinteraction accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent assigns different functional qualities to different parts of the hand. The index finger tip is designated as the primary interaction point for precision operations, while the palm and other fingers serve as the projection surface and support structure. This local differentiation of function enhances both the precision of click operations and the stability of interface projection.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system implements continuous feedback by tracking the relative position between the index fingertip and the open hand, providing real-time information about gesture execution. The virtual interface responds dynamically to gesture movements, confirming recognition and execution of commands, which enhances interaction reliability and user confidence.

Inventive Principle:

Principle #23Feedback

3Ease of operation

If virtual interface is projected on open hand, then interaction becomes more immersive, but tracking precision and command interpretation accuracy may be compromised

Engineering Contradiction:

Improveinteraction immersionVSAvoidfingertip position accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent introduces the relative position relationship between the index fingertip and the open hand as an intermediary measurement framework. Instead of measuring absolute positions, the system tracks the displacement and spatial relationship between these two reference points, which improves measurement precision by providing a stable reference frame that moves with the hand.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10678342B2Method of virtual user interface interaction based on gesture recognition and related device

Publication Date: 2020.06.09 XRSPACE CO LTD

AI summary

A method of virtual user interface interaction based on gesture recognition comprises detecting two hands in a plurality of images, recognizing each hand's gesture, projecting a virtual user interface on an open gesture hand when one hand is recognized with a point gesture and the other hand is recognized with an open gesture, tracking an index fingertip of the point gesture hand, determining whether the index fingertip of the point gesture hand is close to the open gesture hand within a predefined rule, interpreting a movement of the index fingertip of the point gesture hand as a click command when the index fingertip of the point gesture hand is close to the open gesture hand within the predefined rule, and in response to the click command, generating image data with a character object of the virtual user interface object.