Automated GUI Testing via Visual Pattern Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional automation approaches for testing software systems with graphical user interfaces (GUIs) require skilled programmers for setup and maintenance, making the process costly and time-consuming, and are limited to regression testing, achieving positive return on investment only after multiple releases.
Innovation Solution
An automated testing system that uses optical character recognition (OCR), pattern matching, and visual detection algorithms to learn and interact with GUIs without requiring access to programming code, allowing non-programmers to create and maintain test cases by identifying and associating text elements with user input objects and retrieving test parameter values from a database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If traditional automation approaches are used for testing software systems with GUIs, then testing can be automated, but skilled programmers are required for setup and maintenance, making the process costly and time-consuming
Solution Approach 1:
The patent replaces traditional code-based test automation mechanisms with a visual, image-based interaction system. Instead of requiring programmers to write code that programmatically interacts with GUI elements, the system uses image recognition and visual matching to automatically identify and interact with GUI components. This substitution eliminates the need for programming expertise while maintaining automation capabilities.
Solution Approach 2:
The patent introduces an intermediary layer between the test automation system and the GUI under test. This intermediary uses image capture and pattern matching to mediate the interaction, translating visual GUI elements into actionable test steps without requiring direct code-based access to the application's internal structure. This intermediary abstraction simplifies the automation process and reduces complexity.
2Productivity
If traditional automation approaches are used, then testing can be performed, but the setup occurs prior to test execution and is very costly
Solution Approach 1:
The patent performs preliminary actions by automatically capturing and analyzing GUI images during the test setup phase, pre-identifying all relevant GUI elements and their locations. This preliminary visual analysis creates a map of the GUI structure that can be reused across multiple test executions, eliminating the need for repeated setup procedures and reducing both time and cost.
Solution Approach 2:
The patent creates visual copies or representations of the GUI interface through image capture and analysis. These visual models serve as templates that can be reused across multiple test cases and executions, eliminating the need to re-program or re-configure test setups for each new test run, thereby significantly reducing setup time and costs.
3Reliability
If traditional automation approaches are used, then test cases can be executed, but maintenance of these tests is a very costly process requiring coding updates
Solution Approach 1:
The patent replaces code-based test maintenance with visual image-based maintenance. When GUI elements change, the system automatically detects these changes through image comparison and updates its visual models accordingly, eliminating the need for programmers to manually update code. This maintains test reliability while dramatically improving ease of maintenance.
Solution Approach 2:
The patent enables the test automation system to self-maintain by automatically detecting changes in the GUI interface through image analysis and autonomously updating its internal visual models. This self-service capability eliminates the need for external programmer intervention for routine maintenance tasks, reducing costs while maintaining reliability.
4Extent of automation
If traditional automation approaches are used, then testing can be automated, but only programmers can staff testing projects
Solution Approach 1:
The patent substitutes code-based operation with visual, image-based interaction. Testers can define and execute test cases by working with visual representations of the GUI rather than writing or modifying code. This substitution makes test automation accessible to non-programmers while maintaining full automation capabilities.
Solution Approach 2:
The patent introduces a visual intermediary layer that translates between simple user-friendly test definitions and the complex automation operations required to execute them. This intermediary allows testers with no programming knowledge to create and maintain automated tests by working with visual GUI elements rather than code.
Data Source
AI summary
A system for automatically testing an application system graphical user interface (GUI) includes first and second application servers communicatively connected across a communication network. The second application server retrieves a GUI page provided by the first application server as part of the application system, and identifies text elements and user input objects in an image of the retrieved GUI page. Each user input object is then associated with a text element. Test parameter values are retrieved from a database storing test parameter data, and the application system is tested. In particular, for each user input object of the GUI page, a respective test parameter value is provided that is associated in the database with a same text element as is associated with the user input object. A response of the application system is then monitored.