Virtual Keyboard Control for HMDs Using Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile terminals, such as head-mounted displays (HMDs), face challenges in providing intuitive manipulation inputs as they lack physical keyboards or mice, leading to user burden and difficulty in precise text input with gestures or sound inputs.
Innovation Solution
A control device and storage medium that detect a substantial object's manipulation region and associate a predetermined function with it, allowing users to perform inputs by recognizing the positional relation between the object and the manipulator, enabling the use of real-world objects as virtual input devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If head-mounted displays (HMDs) use gesture or sound inputs for manipulation, then the device achieves hands-free operation, but the manipulation precision and ease of text input deteriorates
Solution Approach 1:
The patent introduces an intermediary system consisting of a projector and screen that creates a virtual keyboard in the user's field of view. The projector displays virtual keys on a screen, and the system detects the user's hand position and gestures to interact with these virtual keys. This intermediary interface bridges the gap between hands-free operation and precise text input by providing visual feedback and structured input zones.
Solution Approach 2:
The patent replaces the mechanical keyboard system with an optical and computational system. Instead of physical keys that require tactile feedback, the system uses a projector to display virtual keys, cameras or sensors to detect hand gestures, and computational algorithms to recognize and interpret the gestures as text input commands. This substitution enables hands-free operation while maintaining input precision through visual and gestural interfaces.
2Device complexity
If HMDs rely solely on gesture or sound manipulations, then the device structure is simplified, but the manipulation burden and user stress increases
Solution Approach 1:
The patent creates a universal input interface that can handle multiple types of interactions through a single system. The virtual keyboard can accommodate various gesture types (pointing, typing motions, swiping), voice commands, and even traditional controller inputs. This multi-functional approach allows the system to adapt to different user preferences and situations, reducing manipulation burden by providing flexible input options rather than requiring users to master a single complex gesture system.
Solution Approach 2:
The patent dynamically adjusts parameters of the virtual keyboard interface based on user behavior and context. The system can change key sizes, spacing, and layout based on the detected gesture type, user distance from the screen, and environmental conditions. This adaptability reduces manipulation burden by optimizing the interface for each specific interaction scenario, making the system easier to use without requiring structural complexity.
3Ease of operation
If virtual manipulation regions are associated with substantial objects, then the ease of operation improves, but the device complexity increases
Solution Approach 1:
The patent creates a virtual copy of a keyboard interface that is projected onto or near a substantial object (such as a physical desk or table). This virtual keyboard copy maintains the familiar layout and structure of a traditional keyboard, allowing users to intuitively understand and use the interface. The system detects the position and orientation of the substantial object and adjusts the virtual keyboard accordingly, providing intuitive manipulation while using relatively simple projection and detection technologies.
Data Source
AI summary
According to the present disclosure, there is provided a control device including a detection unit configured to detect, as a manipulation region, at least a part of a substantial object present at a position at which a user is estimated to be able to perform a manipulation, a function setting unit configured to perform setting in a manner that a predetermined function matches the manipulation region detected by the detection unit, and a control unit configured to perform the function matched with the manipulation region based on a positional relation between the manipulation region and a manipulator.