Automated Keyboard Layout Generation from Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The creation of virtual keyboard layouts for computing devices with touch sensors is a time-consuming and error-prone process, especially when converting unstructured images into structured keyboard layout definitions that can be deployed on other devices, as existing methods struggle to accurately extract key positions, labels, and actions from visual representations.
Innovation Solution
A system and method for automated structured keyboard layout generation, which involves obtaining an image of a keyboard layout, detecting key boundaries, determining label strings, selecting corresponding key actions, and generating a keyboard layout definition including key positions, labels, and actions, using image processing and OCR techniques, with optional machine learning for complex icons, to create a deployable definition for mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual creation of virtual keyboard layouts is used, then accuracy and customization can be maintained, but the process is time-consuming and error-prone
Solution Approach 1:
The patent replaces manual mechanical processes of keyboard layout creation with an automated image processing system. The system uses computer vision algorithms to detect key boundaries, OCR (optical character recognition) to extract labels, and automated mapping to generate structured keyboard layout definitions, eliminating the need for manual transcription and reducing errors associated with manual creation
Solution Approach 2:
The system creates a digital copy of the physical or reference keyboard layout image and processes this copy through automated algorithms. By working with an image representation rather than manually transcribing each key, the system efficiently extracts all key positions, labels, and actions while maintaining accuracy through automated detection and recognition processes
2Productivity
If automated image processing is used to extract keyboard layouts, then time consumption is reduced, but extraction accuracy and reliability deteriorate
Solution Approach 1:
The patent divides the keyboard layout image into discrete key regions by detecting key boundaries first. This segmentation allows the system to process each key independently, applying OCR and action mapping to specific regions rather than the entire image, which improves both processing speed and accuracy by reducing computational complexity and enabling targeted analysis of each key element
Solution Approach 2:
The system incorporates validation mechanisms where extracted key positions, labels, and actions are checked against expected patterns and constraints. The automated process includes verification steps that compare extracted information against the original image and against each other, allowing correction of extraction errors and ensuring reliability while maintaining high productivity through automated validation
3Adaptability or versatility
If complex icons and labels are present in keyboard images, then keyboard functionality is enhanced, but automated recognition and extraction become more difficult
Solution Approach 1:
The patent applies different processing techniques to different regions of the keyboard image based on their content characteristics. Text labels are processed using OCR algorithms, while icons and symbols are processed using pattern recognition and template matching. This localized approach allows the system to handle diverse keyboard elements (text, icons, numbers, special characters) with appropriate specialized algorithms for each type, maintaining versatility while managing recognition difficulty
Solution Approach 2:
The system uses a universal processing framework that can handle multiple types of keyboard elements through a single integrated pipeline. The same automated system processes text labels, numeric characters, symbolic icons, and special characters by routing them through appropriate recognition modules, enabling the keyboard to support diverse layouts and functionalities without requiring separate processing systems for each element type
Data Source
AI summary
A method includes: obtaining an image of a keyboard layout; detecting, from the image, a plurality of key boundaries; determining a label string for a portion of the image defined by a corresponding key boundary from the plurality of key boundaries; selecting, for the determined label string, a corresponding key action; generating a keyboard layout definition for the image of the keyboard layout, the keyboard layout definition including a key definition for the corresponding key boundary, the key definition having: (i) a position for rendering of a key, (ii) a label configured to be rendered at the position of the key, and (iii) an action configured to be caused by selection of the key; and communicating the keyboard layout definition for deployment to a mobile device.