Transparent Input Device Hand Tracking via Image Sensor
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users of multi-touch input devices, such as tablets and smartphones, face challenges in tracking finger positions due to the need to look at a display oriented differently, leading to ergonomic strain and loss of touch screen versatility when using external pointing devices.
Innovation Solution
An input device with an image sensor and partially transparent input element captures hand images from below, processing them to identify anatomical features and provide visual feedback, preserving spatial relationships between the physical and virtual worlds, allowing precise tracking of fingers and stylus points without external cameras.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If users look at a display oriented differently from the input device, then the display can be positioned for viewing comfort, but the user loses track of finger positions and experiences ergonomic strain
Solution Approach 1:
The system provides visual feedback by overlaying the processed hand image with anatomical feature indicators directly on the display, allowing users to see their hand position and the corresponding cursor position simultaneously. This feedback loop resolves the contradiction by maintaining display viewing comfort while preventing loss of finger position tracking information.
Solution Approach 2:
The processed hand image acts as an intermediary between the physical hand and the virtual cursor on the display. By mapping anatomical features to cursor positions through this intermediate representation, the system bridges the spatial gap between the input device and display, allowing users to maintain both viewing comfort and position awareness.
2Ease of operation
If a physical keyboard and external display are connected to a tablet or smartphone, then data entry ergonomics are improved, but the user loses sight of the device display and multi-touch capability
Solution Approach 1:
The system enables the touch screen to serve multiple functions simultaneously: it acts as both the display and the input device. By processing hand images captured through the touch screen and providing visual feedback on the same screen, the system maintains multi-touch capability while improving data entry ergonomics, eliminating the need for separate external devices.
3Ease of operation
If an external pointing device such as a mouse is used, then typing facilitation is improved, but the user loses correspondence between finger location and action position on screen
Solution Approach 1:
The system creates a visual copy of the hand image and overlays it on the display with corresponding anatomical features mapped to cursor positions. This copy maintains the spatial correspondence between physical finger location and virtual action position, providing typing facilitation without losing the intuitive connection between hand position and screen interaction.
Data Source
AI summary
A method (62) for providing visual feedback to a user having a hand (14) and operating a graphical user interface (28) on a display (24) using an input device (10). The input device (10) includes an image sensor (46) and an input element (40) defining an input surface (42), the input element (40) being at least partially transparent. The method (62) includes: with the hand (14) substantially adjacent to the input element (40), acquiring a hand image of the hand (14) through the input element (40) using the image sensor (46); identifying predetermined anatomical features (13) in the hand image and associate with each a corresponding feature location indicative of a position of the predetermined anatomical features (13) relative to the input surface (40); and displaying cursors (34, 61) in a window (30) of the graphical user interface (28) at positions relative to the window (30) that correspond to the feature locations relative to the input surface (40).