Automated Visual Testing via Masked UI Element Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional visual testing tools are inadequate for accurately testing software applications' graphical user interfaces (GUIs) due to their inability to effectively handle screenshots, which do not change in field of view or luminance, leading to repetitive and error-prone manual testing with inconsistent results.
Innovation Solution
An automated visual testing system that captures images of the application, applies masks to filter out time-variant and version-specific changes, and compares them with predetermined patterns using perception statistics to reduce false positives, employing selective-alpha and ignore-bitmap masks, and test stabilization to handle non-deterministic tests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing visual testing tools (e.g., PerceptualDiff) are used to detect differences between screenshots, then field of view, Gamma and luminance changes can be detected, but the tools cannot effectively and accurately visually test applications because screenshots do not change these parameters
Solution Approach 1:
The patent changes the testing parameters from field of view, Gamma and luminance (which do not change in screenshots) to color values of specific UI elements. The system captures screenshots, locates UI elements using coordinates, and compares color values at those locations between expected and actual screenshots, making the testing parameters relevant to screenshot comparison.
Solution Approach 2:
The patent creates a copy of the expected screenshot and uses it as a reference for comparison. The system captures an expected screenshot, stores it, and then compares actual screenshots against this stored copy by comparing color values of UI elements at corresponding locations, enabling automated visual validation.
2Reliability
If manual visual testing is performed by eye-balling screenshots, then the GUI layout and content can be visually validated, but the process is repetitious, error-prone and produces inconsistent results when performed by different people
Solution Approach 1:
The system performs automated self-testing by programmatically capturing screenshots, locating UI elements using predefined coordinates, extracting color values, and comparing them against expected values. This eliminates the need for manual eye-balling by different people, ensuring consistent and reproducible results while reducing testing time.
Solution Approach 2:
The patent replaces the manual mechanical process of eye-balling screenshots with an automated computational system. The system uses programming to capture screenshots, locate UI elements by coordinates, extract color values, and perform automated comparison, substituting human visual inspection with machine-based automated testing.
3Extent of automation
If automated visual testing is implemented using existing tools, then the testing process can be automated, but false positives increase because the tools are not suited for screenshot comparison
Solution Approach 1:
Instead of performing global image comparison, the patent focuses on local color value comparison at specific UI element locations. The system uses predefined coordinates to locate specific UI elements and compares color values only at those locations, rather than comparing entire screenshots globally, thereby improving detection accuracy and reducing false positives.
Solution Approach 2:
The patent segments the screenshot comparison process into discrete UI element comparisons. Each UI element is located using predefined coordinates, and its color value is compared independently. This segmentation allows precise tracking of specific UI element changes and reduces false positives caused by global image differences.
Data Source
AI summary
Some embodiments of a system and a method to automate visual testing of an application have been presented. For instance, images generated by the application under test can be automatically captured. To reduce false positives in the automated visual testing, masks are applied onto the images to filter out some predetermined parts of the images. Then the masked images are compared with a set of predetermined patterns. Differences between the masked images and the predetermined patterns may be reported.