Virtual Mouse XR Control via Hand Gesture and IMU Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing extended reality (XR) interaction methods, such as hand gestures and remote controllers, face issues like occlusion problems, user fatigue, and inconvenience, particularly in augmented reality (AR) systems, where hand-free interaction is required.
Innovation Solution
A system and method for remotely controlling XR environments using a virtual mouse, where a head-mounted display (HMD) detects specific hand gestures to switch control modes and enables a virtual mouse simulation using ring-type remote controllers with IMUs and optical sensors, allowing intuitive interaction with XR scenes through move, click, and scroll operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If hand gesture is used to interact with XR environment, then user can perform natural interaction, but occlusion issues cause unexpected input and hand fatigue occurs
Solution Approach 1:
The patent introduces an optical sensor as an intermediary device that captures images of the user's hand and processes them to generate control signals. This mediator translates natural hand gestures into reliable digital commands, eliminating occlusion issues while preserving gesture-based interaction naturalness.
Solution Approach 2:
The patent replaces the mechanical hand gesture recognition system (which suffers from occlusion) with an optical sensing system using cameras and image processing. This substitution maintains the natural interaction paradigm while eliminating the reliability problems of direct gesture recognition.
2Measurement precision
If remote controller is used to interact with XR environment, then control precision can be achieved, but user cannot be hand free and mapping relationship is complex
Solution Approach 1:
The system uses the user's own hand as the control interface. The optical sensor captures images of the hand, and the processor automatically identifies hand posture and generates corresponding control signals. This self-service approach eliminates the need for external remote controllers while maintaining precise control through sophisticated hand gesture recognition.
3Ease of operation
If virtual mouse is enabled through hand posture recognition, then intuitive control can be achieved, but system complexity increases
Solution Approach 1:
The patent makes the HMD device multi-functional by integrating both XR display capabilities and hand gesture recognition capabilities into a single device. The optical sensor and processor within the HMD can detect various hand postures and translate them into different control operations (cursor movement, clicking, scrolling), providing universal control for multiple functions without requiring separate devices.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Provides a convenient and intuitive way for users to interact with XR environments, reducing fatigue and occlusion issues while enabling seamless hand-free control, enhancing usability in AR and VR systems.
Implementation Method 1
a remote controller (210) worn on by the user's hand (500), wherein the remote controller (210) includes one or more inertial measurement units (IMU)
Implementation Method 2
US 2015/0205360 A1 discloses a method, including: capturing, using an image sensor of an information handling device, a user gesture input
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A system and a method for remotely controlling extended reality by a virtual mouse are provided. The system includes a head mounted display and a first remote controller for being worn on a hand. The head mounted display includes an image capture device. The first remote controller communicatively connects to the head mounted display, wherein the head mounted display is configured to: capture an image through the image capture device; detect the image to obtain first data of the hand in the image and second data of a plane in the image; and enable the virtual mouse according to the first data and the second data.