GUI Object Recognition for Automated Software Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated software testing tools face challenges in reliability and maintenance, particularly when updating tests for applications, as they often require manual coding and struggle with recognizing graphical user interface (GUI) objects, leading to increased development time and costs.
Innovation Solution
A computer-based software testing method that presents an interactive visual user interface with nonnative pointing-device-based user interaction, allowing users to intercept and classify GUI objects, build object definitions, and create test paths, thereby simplifying the development and updating of software tests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual coding is used to create test scripts, then test reliability is improved, but test development time and complexity increase
Solution Approach 1:
The system performs preliminary actions by automatically recognizing and classifying GUI objects before test script creation. The object recognition module pre-processes the application interface, identifies control elements, and generates object definitions that are stored for subsequent test development, eliminating the need for manual object identification during testing.
Solution Approach 2:
The system enables self-service by allowing automated generation of test scripts through object recognition and classification. The test script generation module automatically creates test scripts based on recognized GUI objects and their properties, reducing dependency on manual coding while maintaining test reliability through systematic object classification.
2Ease of operation
If recorders are used to capture user activity, then ease of use is improved, but script reliability and maintainability deteriorate
Solution Approach 1:
The system segments the test creation process into distinct modules: object recognition, object classification, test script generation, and test execution. Each module handles specific aspects of test development, allowing for better control and maintenance. The object recognition module separately identifies GUI elements, while the classification module organizes them into structured definitions, improving both reliability and maintainability.
Solution Approach 2:
The system introduces an intermediary layer of object definitions between user actions and test scripts. Instead of directly recording user actions, the system captures GUI object properties and relationships, creating a structured intermediate representation that enhances script reliability and ease of maintenance.
3Productivity
If automated object recognition is implemented, then test development speed is improved, but system complexity increases
Solution Approach 1:
The system implements universal object recognition capabilities that can identify various types of GUI controls (buttons, text fields, lists, tables) using a unified approach. The object recognition module serves multiple functions: identifying control elements, extracting their properties, and generating standardized object definitions, thereby improving productivity without proportionally increasing complexity.
Solution Approach 2:
The system creates simplified copies of GUI objects in the form of object definitions that capture essential properties and relationships. These object definitions are lightweight representations that enable automated test script generation without requiring the full complexity of the original application interface, thus improving development speed while managing system complexity.
4Reliability
If comprehensive test coverage is achieved, then application quality is improved, but test maintenance effort increases
Solution Approach 1:
The system performs preliminary classification and organization of GUI objects into structured definitions before comprehensive testing begins. This pre-organization enables easier maintenance by providing a clear hierarchy and relationships between test objects, reducing the effort required to update tests when applications change.
Solution Approach 2:
The system implements feedback mechanisms that track object relationships and dependencies within the application interface. When applications are updated, the system can automatically detect changes in object definitions and notify users of affected test scripts, thereby maintaining comprehensive coverage while reducing manual maintenance effort.
Data Source
AI summary
In one general aspect, a computer-based software testing method is disclosed that includes presenting to a user an interactive visual user interface of an application under test that includes a plurality of user interface controls, with each of the presented user interface controls having a predefined native interaction mode. The method also includes intercepting normative pointing-device-based user interaction with each of a plurality of the user interface controls, with the normative pointing-device-based user interaction mode being different from the native interaction mode. Interface definitions are built for the application based on results of the steps of intercepting for the plurality of interface controls.