Camera-Based Virtual Keyboard Input with Finger Position Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing virtual keyboards on electronic devices suffer from high typographical error rates due to uniform detection of hand movements without considering device posture and finger position and bending, making it difficult to distinguish between keys in the same column and adjust the keyboard layout.

Innovation Solution

An electronic device with a camera and processor that crops images to detect finger objects, matches key columns to image regions, identifies key rows based on finger positions and bending, and determines virtual key inputs through typing movements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If uniform detection of hand movement is used without considering device posture and finger position, then the detection process is simple, but the typographical error rate increases

Engineering Contradiction:

Improvedetection processVSAvoidtypographical error rate

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The system changes detection parameters dynamically based on device posture (horizontal/vertical) and finger characteristics (position, bending degree). Instead of using fixed detection parameters, the system adjusts key column and key row determination parameters according to the detected device orientation and finger state, thereby reducing typographical errors without significantly increasing system complexity

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The detection system transitions from a static uniform detection approach to a dynamic adaptive detection approach. The system continuously adapts detection parameters based on real-time device posture and finger position, making the detection process flexible and context-aware while maintaining reasonable complexity

Inventive Principle:

Principle #15Dynamics

2Stability of the object's composition

If key columns are fixedly matched without considering finger position and bending, then the keyboard layout is stable, but the ability to distinguish keys in the same column is reduced

Engineering Contradiction:

Improvekeyboard layoutVSAvoidkey identification accuracy

Core Design Contradiction:

Stability of the object's compositionVSMeasurement precision

Solution Approach 1:

The system applies different matching criteria for different regions of the keyboard. Instead of using a single fixed matching rule for all keys, the system determines key columns and key rows using different parameters based on local finger position and bending degree, enabling precise differentiation of keys within the same column while maintaining overall layout stability

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The keyboard detection is segmented into two independent dimensions: key column determination (based on horizontal finger position) and key row determination (based on finger bending degree). This segmentation allows the system to independently optimize each dimension's detection parameters, improving key identification accuracy without compromising layout stability

Inventive Principle:

Principle #1Segmentation

3Device complexity

If device posture is not considered in image processing, then the processing is simpler, but the accuracy of finger position detection is reduced

Engineering Contradiction:

Improveimage processingVSAvoidfinger position detection accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The image processing system dynamically adjusts the cropping region based on detected device posture. When the device is in horizontal or vertical orientation, the system automatically adjusts the image cropping parameters to capture the relevant finger region accurately, thereby improving detection precision without requiring complex processing for all possible orientations

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12429955B2Electronic device for obtaining user input through virtual keyboard and method of operating the same

Publication Date: 2025.09.30 SAMSUNG ELECTRONICS CO LTD

AI summary

Disclosed is an electronic device configured to obtain a second image, by cropping a first image obtained through a camera, detect at least one finger object in the second image, the at least one finger object corresponding to at least one finger, determine at least one key column corresponding to the at least one finger object among a plurality of key columns of a virtual keyboard by matching the plurality of key columns of the virtual keyboard to a plurality of regions of the second image, and identifying at least one region corresponding to at least one position of the at least one finger object among the plurality of regions of the second image, determine at least one key row corresponding to the at least one finger object among a plurality of key rows of the virtual keyboard, identify, based on the determined at least one key column and the determined at least one key row, at least one virtual key corresponding to the at least one finger object among a plurality of virtual keys of the virtual keyboard, and identify an input of the identified at least one virtual key, based on detecting a typing movement of the at least one finger object.