GUI Element Identification via Image Capture for Automated Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automated testing tools face difficulties in identifying graphical user interface (GUI) elements, especially when object hierarchies are not accessible, such as in web development applications like FLASH or through image-based protocols like RDP, making it challenging to generate test scripts without modeling the programming language's object hierarchy.
Innovation Solution
A method that captures images of GUIs in response to user inputs, identifies the relevant GUI elements, and generates test scripts based on image data, allowing for simulated inputs to be provided to applications even when the object hierarchy is not accessible, thereby facilitating application testing without requiring explicit modeling of the programming language's object hierarchy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional automated testing tools model object hierarchies of programming languages to identify GUI elements, then testing accuracy is improved, but device complexity and implementation difficulty increase significantly
Solution Approach 1:
The patent creates visual copies of the GUI interface through screenshot capture and uses image processing techniques to identify GUI elements. Instead of modeling complex object hierarchies, the system captures visual representations of the interface and extracts element information from these images, significantly reducing implementation complexity while maintaining identification accuracy.
Solution Approach 2:
The patent replaces the mechanical system of object hierarchy modeling with an optical/image-based system. By using screenshot capture and image processing algorithms, the system substitutes the traditional approach of traversing programming language object trees with a visual recognition method that is easier to implement and maintain.
2Adaptability or versatility
If automated testing tools access applications through image-based protocols like RDP, then accessibility to various applications is improved, but ability to identify GUI elements deteriorates due to inaccessible object hierarchy
Solution Approach 1:
The patent introduces an image-based intermediary layer between the testing tool and the application. By capturing screenshots through protocols like RDP and processing these images to identify GUI elements, the system creates a mediator that bridges the gap between limited protocol access and comprehensive element identification, allowing testing of applications that would otherwise be inaccessible.
Solution Approach 2:
The system creates visual copies of the remote application interface through screenshot capture. These image copies serve as the basis for GUI element identification, allowing the testing tool to analyze and interact with remote applications even when direct object hierarchy access is unavailable through the protocol.
3Reliability
If conventional testing tools require explicit modeling of programming language object hierarchies, then testing reliability is improved, but ease of operation and implementation deteriorates
Solution Approach 1:
The patent implements self-service by automatically capturing screenshots and processing images to identify GUI elements without requiring manual configuration of object hierarchies. The system autonomously performs the identification task by analyzing visual information, eliminating the need for testers to manually model programming language structures while maintaining reliable element identification.
Solution Approach 2:
The system performs preliminary actions by capturing screenshots and pre-processing images before actual testing operations. This preliminary image capture and analysis creates a visual map of the GUI interface that facilitates subsequent testing operations, reducing the complexity of real-time element identification during test execution.
Data Source
AI summary
Systems, methods, and other embodiments associated with graphical user interface input element identification are described. One example system identifies a graphical user interface (GUI) element through which an input to an application was received based on comparing images generated from data provided by the application. Data that describes the GUI element may then be provided.