Automated GUI Reference Image Generation via Pixel Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software testing across multiple devices and operating systems is time-consuming and prone to errors due to the difficulty in accurately selecting and creating reference images for automated test descriptions, leading to potential test failures and inefficiencies in regression testing.
Innovation Solution
A method that automatically identifies and generates reference data items for graphical user interface (GUI) objects by determining correlations between pixels, creating accurate image boundaries, and storing these in a database for use in updating automated computer test descriptions, thereby reducing manual intervention and improving test reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual selection and creation of reference images is performed, then test coverage can be achieved, but the process is time-consuming and error-prone
Solution Approach 1:
The system performs self-service by automatically generating reference images through pixel correlation analysis without requiring manual intervention. The automated process identifies GUI objects, determines their boundaries, and creates reference data items independently, eliminating the time-consuming manual selection process while maintaining test reliability.
Solution Approach 2:
The manual mechanical process of selecting and creating reference images is replaced with an automated computational system. The patent uses pixel correlation algorithms and automated image processing to substitute the manual mechanical operations, thereby reducing testing time while preserving or improving reliability through consistent automated execution.
2Measurement precision
If automated pixel correlation analysis is performed to identify GUI objects, then reference image accuracy is improved, but processing complexity increases
Solution Approach 1:
The complex processing task is segmented into distinct stages: pixel correlation analysis, GUI object identification, boundary determination, and reference image generation. By dividing the processing into modular segments, the system achieves high measurement precision through systematic analysis while managing complexity through structured decomposition of the overall process.
Solution Approach 2:
The patent introduces intermediate processing steps such as pixel correlation analysis and boundary determination as mediators between the raw GUI screenshot and the final reference image. These intermediary processes systematically transform the input data through defined algorithms, improving accuracy while containing complexity within manageable intermediate stages.
3Adaptability or versatility
If reference images are automatically generated for all GUI objects, then test description completeness is improved, but data storage requirements increase
Solution Approach 1:
The system extracts only the essential and relevant GUI objects that require reference images for testing, rather than generating references for all possible elements. By selectively extracting necessary reference data based on test requirements, the system achieves complete test descriptions while minimizing data storage requirements through targeted extraction of only needed reference images.
Data Source
AI summary
Some embodiments are directed to a method comprising automatically obtaining one or more reference data items representative of one or more graphical user interface (GUI) objects from a GUI of a system under test by, for each of the one or more reference data items: identifying one or more correlations between two or more pixels of the GUI of the system under test to determine a group of pixels corresponding to a GUI object; determining image boundaries of the GUI object based on at least one of the determined group of pixels and any identified correlations; generating a reference data item representative of the GUI object identified by the determined image boundaries; and storing the reference data item in a database. The method also comprises receiving a user input representative of a selection of a desired reference data item from the database and updating a test description for an automated computer test with the selected reference data item.