GUI Input Element Identification via Blob Detection and Grid Intersection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated testing of graphical user interfaces (GUIs) is challenging due to input elements without pre-defined screen locations, leading to difficulties in identifying correct input elements for testing, especially when elements are randomized or surrounded by non-relevant screen elements.
Innovation Solution
A technique involving blob detection analysis to identify potential input elements, forming rows and columns, and determining intersections to accurately identify input elements for automated testing, which includes using optical character recognition (OCR) and target patterns to refine the identification process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual testing is used to identify input elements in GUIs, then testing accuracy is maintained, but testing efficiency and productivity are reduced
Solution Approach 1:
The patent replaces manual mechanical testing operations with an automated computer vision system that uses blob detection algorithms and image processing to automatically identify input elements in GUIs, eliminating the need for human testers to manually locate and interact with screen elements
Solution Approach 2:
The system enables automated testing by having the computer vision algorithm independently identify and locate input elements without human intervention, allowing the testing process to self-execute by automatically detecting blobs, determining their characteristics, and using them as test targets
2Productivity
If automated testing is implemented without pre-defined screen locations, then productivity is improved, but measurement precision of input element identification deteriorates
Solution Approach 1:
The patent replaces traditional coordinate-based automated testing with a blob detection-based system that uses image processing to identify input elements, substituting mechanical coordinate input with automated visual recognition to maintain precision without pre-defined locations
Solution Approach 2:
The system changes the identification parameter from fixed screen coordinates to dynamic blob characteristics (area, shape, position), allowing automated testing to adapt to randomized element locations by detecting and analyzing visual features rather than relying on predetermined positions
3Measurement precision
If blob detection analysis is performed on all screen elements, then comprehensive identification is achieved, but device complexity and processing time increase
Solution Approach 1:
The patent segments the screen into regions of interest by detecting blobs and grouping them based on spatial relationships and visual characteristics, processing only relevant areas rather than analyzing every pixel across the entire screen, thus reducing computational complexity while maintaining comprehensive identification
Data Source
AI summary
A computing device includes a processor and a medium storing instructions. The instructions are executable by the processor to: identify, based on a blob detection analysis, a plurality of potential input elements in a graphical user interface (GUI); determine a set of rows including potential input elements that are in a horizontal alignment and in a same size range; determine a set of columns including potential input elements that are in a vertical alignment and in a same size range; determine a set of input elements comprising multiple potential input elements that are located at intersections of the identified set of rows and the identified set of columns; and perform automated testing of the GUI using the determined set of input elements.