Camera-Based Keyboard Keystroke Localization via Image Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-entry methods for small mobile devices, such as wearable and projection keyboards, face challenges like requiring additional equipment, limited usability, and high latency, particularly in noisy environments or when using finger pads instead of finger nails.
Innovation Solution
A camera-based keyboard system that uses a mobile device's front-facing camera to detect and localize keystrokes by capturing images of a user typing on a paper keyboard, employing image processing techniques for high accuracy and low latency, and optimizing image processing for efficient operation on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If wearable keyboards with additional sensors are used, then text input capability is improved, but device complexity increases
Solution Approach 1:
The patent uses a camera to capture optical images of a physical keyboard and fingertips, creating a visual copy of the typing interaction. This eliminates the need for wearable sensors by replicating the detection function through image processing, thereby maintaining text input capability while reducing device complexity
Solution Approach 2:
The patent replaces mechanical sensor-based detection (accelerometers, pressure sensors in wearable keyboards) with an optical detection system using a camera and image processing algorithms. This substitution eliminates the need for additional mechanical components while preserving the ability to detect keystrokes
2Illumination intensity
If projection keyboards with infrared projectors are used, then keyboard visibility is improved, but device complexity increases
Solution Approach 1:
Instead of projecting a virtual keyboard using infrared or visible light projectors, the patent captures an image of a physical keyboard layout and processes it through image recognition algorithms. This copying approach maintains keyboard visibility and structure while eliminating the need for complex projection equipment
Solution Approach 2:
The patent substitutes the optical projection system (infrared/visible light projectors) with a camera-based capture and image processing system. This replacement maintains the visual guidance function while significantly reducing device complexity by using standard camera hardware instead of specialized projection equipment
3Difficulty of detecting and measuring
If UbiK audio-based localization is used, then keystroke detection is improved, but reliability decreases in noisy environments
Solution Approach 1:
The patent replaces the acoustic detection method (microphone-based keystroke localization) with an optical detection method using a camera. This substitution eliminates susceptibility to acoustic noise while maintaining the ability to detect and localize keystrokes through visual image processing of fingertip positions on keyboard keys
4Ease of operation
If traditional on-screen keyboards are used, then text input is enabled, but screen area is reduced for other functions
Solution Approach 1:
The patent moves the keyboard interface from the two-dimensional screen space to the physical three-dimensional space by using a camera to capture a physical keyboard. This allows the keyboard to exist outside the screen boundaries, preserving full screen area for other functions while maintaining text input capability through optical detection of physical key presses
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables accurate and efficient text input with all fingers, similar to a traditional keyboard, reducing false positives and time latency, and is adaptable for various environments without the need for additional equipment.
Implementation Method 1
capture a plurality of images regarding the keyboard and at least one hand typing on the keyboard via the camera
Data Source
AI summary
Systems, methods and techniques are provided for interacting with mobile devices using a camera-based keyboard. The system comprises a processor system including at last one processor. The processor system is configured to at least capture a plurality of images in connection with the keyboard and at least one hand typing on the keyboard via the camera. Based on the plurality of captured images, the processor system is further configured to locate the keyboard, extract at least a portion of the keys on the keyboard, extract a hand, and detect a fingertip of the extracted hand. After that, a keystroke may be detected and localized through tracking the detected fingertip in at least one of the plurality of captured images, and a character corresponding to the localized keystroke may be determined.