Handheld Device Hover Detection via Mono-Chromatic Overlay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In augmented and virtual reality systems, users face difficulties interacting with handheld electronic devices while immersed in a virtual environment due to the challenge of viewing and manipulating user interfaces, such as keyboards, which are obscured by the virtual environment displayed on head-mounted devices.
Innovation Solution
A method involving a head-mounted device that captures images of a handheld electronic device with a mono-chromatic screen overlaid on its user interface, allowing the camera to detect and track the user's finger position, and virtually render the interface and object in the virtual environment, maintaining touchscreen functionality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a head-mounted device displays a virtual environment, then the user experiences an immersive 3D virtual environment, but the user cannot directly view the handheld device's screen to interact with user interfaces
Solution Approach 1:
The system creates a virtual copy of the handheld device's user interface within the virtual environment displayed by the head-mounted device. This virtual representation allows users to interact with UI elements without needing to see the physical screen, resolving the contradiction between immersive experience and ease of operation
Solution Approach 2:
The system introduces an intermediary mechanism where the head-mounted device's camera captures images of the physical handheld device, and software processes these images to create a virtual representation. This intermediary process enables interaction with the physical device through the virtual environment
2Measurement precision
If a mono-chromatic screen is overlaid on the handheld device display to enable camera detection, then finger position can be detected accurately, but the touchscreen interface becomes invisible to the user
Solution Approach 1:
Instead of displaying the actual screen content directly, the system creates a virtual copy of the user interface within the virtual environment. This copy maintains visibility for the user while allowing the mono-chromatic overlay to function for detection purposes
Solution Approach 2:
The mono-chromatic overlay is applied locally to specific regions of the display where detection is needed, while the virtual environment provides the overall visual context. This localized approach maintains detection capability without completely obscuring the interface
3Measurement precision
If the camera captures images to detect physical objects and finger position, then interaction accuracy improves, but the system complexity increases
Solution Approach 1:
The head-mounted device's camera serves multiple functions: capturing the virtual environment, detecting the handheld device position, and tracking finger movements. This multi-functionality reduces the need for separate detection systems, managing complexity while maintaining precision
Data Source
AI summary
A system for detecting and tracking a hover position of a manual pointing device, such as finger(s), on a handheld electronic device may include overlaying a rendered mono-chromatic keying screen, or green screen, on a user interface, such as a keyboard, of the handheld electronic device. A position of the finger(s) relative to the keyboard may be determined based on the detection of the finger(s) on the green screen and a known arrangement of the keyboard. An image of the keyboard and the position of the finger(s) may be rendered and displayed, for example, on a head mounted display, to facilitate user interaction via the keyboard with a virtual immersive experience generated by the head mounted display.