Combobox Selection Testing Using GUI Difference Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated testing technologies struggle to effectively test the behavior of expandable graphical components in applications, particularly when scrolling through expanded regions, leading to undetected errors and potential propagation of faults to users.
Innovation Solution
A method and system that captures images before and after input events to graphical user interface elements, generates a difference image, and identifies the expanded region of graphical components using optical character recognition and image processing, assigning confidence scores to determine associated text regions, aligning and filtering them for automated testing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If code insertion technology is used to test expandable graphical components, then automated testing capability is improved, but the application may not detect real user actions causing undetected errors
Solution Approach 1:
The patent introduces an intermediary image capture and comparison mechanism between the automated testing system and the expandable graphical component. Instead of directly interacting with the component through code injection, the system captures images of the component before and after user actions, compares these images to detect changes, and uses this visual feedback to verify that selections were properly detected by the application, thereby ensuring reliable error detection while maintaining automation.
Solution Approach 2:
The patent replaces the mechanical code insertion and direct manipulation approach with an optical imaging and image processing system. By using image capture devices to photograph the graphical component and comparing image differences, the system substitutes traditional software-based testing mechanisms with a visual-based approach that can reliably detect whether the application properly responds to user actions on expandable components.
2Reliability
If scrolling through expanded region is required to test expandable components, then testing completeness is improved, but current UFT and device replay cannot effectively find items in the expanded region
Solution Approach 1:
The patent creates visual copies (images) of the expanded region's content and uses image comparison techniques to identify and locate items within the expanded region. Instead of requiring the testing system to physically scroll through and interact with each item, the system captures an image of the expanded region, processes this image copy to identify text regions and items, and uses this information to verify testing completeness without manual scrolling operations.
Solution Approach 2:
The patent transitions from one-dimensional sequential scrolling through expanded regions to a two-dimensional image-based approach. By capturing the entire expanded region as an image and processing it spatially, the system can simultaneously analyze all visible items in the expanded region rather than navigating through them sequentially, greatly simplifying the testing operation while maintaining completeness.
Data Source
AI summary
A method includes capturing a first image of the GUI at a first time, after the first time, providing the GUI with an input event to change a configuration of at least one of the plurality of graphical elements to include an expanded region and capturing a second image of the GUI at a second time after the input event. A background of the second image of the GUI changes from a background of the first image of the GUI to include one or more background text blocks and an expanded region text block. The method also includes obtaining, a difference image, determining whether the difference image includes the one or more background text blocks and the expanded region text block; and selecting a text block closest to a position of the at least one of the plurality of graphical elements as the expanded region text block.