Virtual Mouse for Extended Reality via Hand Gesture Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing extended reality (XR) interaction methods, such as hand gestures and remote controllers, face issues like occlusion problems and user fatigue, and are inconvenient for augmented reality (AR) systems.

Innovation Solution

A system and method for remotely controlling XR using a virtual mouse, where a head-mounted display (HMD) with an image capture device detects hand gestures and enables a virtual mouse mode, allowing users to interact with XR environments through a remote controller worn on the hand, enabling intuitive and convenient interaction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If hand gesture is used to interact with XR environment, then user can perform natural interaction, but occlusion issues cause unexpected input and hand fatigue occurs

Engineering Contradiction:

Improveinteraction convenienceVSAvoidinput accuracy

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent introduces an image capture device as an intermediary to capture the user's hand gestures from the real world and project them into the XR environment. This mediator resolves the occlusion issue by separating the physical hand from the virtual interaction space, allowing accurate gesture recognition without blocking the user's view of the XR content.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If remote controller is used to interact with XR environment, then interaction precision can be improved, but user cannot be hand free and mapping relationship establishment is inconvenient

Engineering Contradiction:

Improvecontrol precisionVSAvoidoperation convenience

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

The system allows the user's own hand to serve as the control interface. By capturing and tracking the user's natural hand gestures through the image capture device, the system eliminates the need for external remote controllers while maintaining precise control. The hand itself becomes both the operator and the controlled interface, providing both precision and freedom.

Inventive Principle:

Principle #25Self-service

3Adaptability or versatility

If traditional remote controller is used, then control function can be achieved, but it is not suitable for augmented reality system where hand freedom is needed

Engineering Contradiction:

Improvesystem compatibilityVSAvoidhand freedom

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent replaces the mechanical remote controller with an optical-based gesture recognition system. The image capture device captures visual information of hand gestures and the processor translates these visual inputs into control commands, substituting mechanical interaction with optical-field-based interaction that is naturally suited for AR environments.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS12147614B2System and method for remotely controlling extended reality by virtual mouse

Publication Date: 2024.11.19 HTC CORP

AI summary

A system and a method for remotely controlling extended reality by a virtual mouse are provided. The system includes a head mounted display and a first remote controller for being worn on a hand. The head mounted display includes an image capture device. The first remote controller communicatively connects to the head mounted display, wherein the head mounted display is configured to: capture an image through the image capture device; detect the image to obtain first data of the hand in the image and second data of a plane in the image; and enable the virtual mouse according to the first data and the second data.