Virtual Mouse via Keyboard Tactile Feedback for VR Input
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
User interfaces for virtual reality (VR) and augmented reality (AR) remain confusing for users, requiring steep learning curves due to the lack of intuitive input methods that effectively utilize hand movements for interacting with virtual environments, especially when physical keyboard access is obstructed.
Innovation Solution
A system and method that enables a virtual mouse functionality on a physical keyboard, using tracking information from sensors like cameras to adjust cursor speed and acceleration, ensuring that user fingers can naturally interact with virtual interface elements by translating key presses into mouse clicks, thereby providing tactile feedback and simplifying user input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional VR/AR user interfaces are used, then visual feedback is provided, but the user interface becomes confusing and requires steep learning curves
Solution Approach 1:
The system segments the input method by separating hand tracking from keyboard interaction. The hand tracker monitors hand position and gesture in 3D space, while the keyboard provides tactile feedback for confirmation. This segmentation allows users to intuitively interact with VR/AR interfaces using natural hand movements while receiving familiar tactile feedback from physical key presses, reducing the learning curve.
Solution Approach 2:
The system introduces an intermediary layer that translates physical keyboard presses into virtual hand gestures. When a user presses a physical key, the system generates the corresponding hand gesture in the virtual environment, allowing users to leverage familiar keyboard muscle memory while interacting with VR/AR interfaces, thereby improving ease of operation.
2Ease of operation
If hand tracking is used to control cursor movement, then tactile feedback is lost, but natural hand movement interaction is enabled
Solution Approach 1:
The system merges hand tracking technology with physical keyboard functionality. The hand tracker captures natural hand movements in 3D space to control cursor position, while the physical keyboard simultaneously provides tactile feedback when keys are pressed. This combination allows users to benefit from both natural gesture-based interaction and the satisfying tactile response of physical keys, eliminating the need to choose between the two approaches.
Solution Approach 2:
The system implements a feedback loop where physical keyboard presses are detected and translated into corresponding virtual hand gestures. When a user presses a physical key, the system generates the appropriate hand gesture signal in the virtual environment, providing immediate tactile feedback that confirms the input while maintaining natural hand movement control. This feedback mechanism bridges the gap between visual hand tracking and tactile keyboard response.
3Reliability
If keyboard keys are mapped to virtual mouse clicks, then tactile feedback is restored, but key positioning precision must be maintained
Solution Approach 1:
The system performs preliminary mapping between physical keyboard keys and virtual hand gesture positions. During system initialization or calibration, the mapping between key positions and corresponding virtual gesture coordinates is pre-established and stored. This preliminary action ensures that when users press physical keys, the corresponding virtual hand gestures are generated with high positional accuracy, maintaining input reliability without requiring real-time complex calculations.
Solution Approach 2:
The system dynamically adjusts mapping parameters based on detected hand position and gesture context. When the user's hand is detected in a specific position or performing a particular gesture, the system modifies the keyboard-to-gesture mapping parameters to optimize alignment accuracy. This parameter adjustment ensures high measurement precision across different hand positions and interaction scenarios, maintaining reliable input accuracy throughout the interaction session.
Data Source
AI summary
Various embodiments include methods for receiving user inputs to a computing device via a virtual mouse function. Methods may include tracking movements of a user's hand, moving a cursor of a virtual mouse in a graphical user interface (GUI) responsive to hand movements, predicting a target user interface element for the cursor in the GUI, and adjusting a speed of the cursor in the GUI based on a distance to the predicted target user interface element so that fingers of the user's hand will be on or over particular keys on the physical keyboard when the cursor is on the predicted target user interface element. Methods may further include switching from a normal operating mode to the virtual mouse operating mode in response to recognizing a user gesture. In the virtual mouse operating mode, key presses on the physical keyboard are translated into mouse click inputs.