GUI Testing Script Generation via Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GUI testing methods for terminal devices with different hardware specifications are labor-intensive and require manual operation, and existing automatic testing techniques are inefficient, particularly in handling changes in touch screen size and resolution, and lack support for physical button testing.
Innovation Solution
A testing device and method that records operations on terminal devices to generate a testing script, combining coordinate-based and object-based techniques, allowing for cross-platform testing and physical button support by detecting actions, reading object information, and recording coordinate and call commands to create a script file.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual testing is used to ensure GUI operations work on different hardware specifications, then testing accuracy can be maintained, but labor consumption and time consumption increase significantly
Solution Approach 1:
The testing device automatically performs testing operations through recorded scripts without requiring manual intervention for each test case. The system records user operations and automatically replays them across different terminal devices, enabling the system to test itself and eliminating the need for continuous manual testing while maintaining comprehensive test coverage
Solution Approach 2:
The system creates a copy of the testing script from recorded operations and uses this script to automatically execute the same testing sequence on multiple terminal devices with different hardware specifications. This copying mechanism allows the tested application to be evaluated across various devices without requiring manual re-testing, thereby maintaining testing accuracy while significantly improving efficiency
2Ease of manufacture
If coordinate-based technique is used for automatic testing, then operation commands are easy to compose, but operation commands must be re-composed when touch screen size or resolution changes
Solution Approach 1:
The system performs preliminary recording of user operations on a reference terminal device to generate a testing script. This pre-recorded script serves as a template that can be adapted to other terminal devices. By capturing the sequence of operations and object interactions in advance, the system establishes a reusable testing framework that reduces the need for manual script composition while maintaining adaptability across different devices
Solution Approach 2:
The system adjusts testing parameters such as coordinate values and object identifiers in the recorded script to match the specific characteristics of different terminal devices. By dynamically modifying these parameters based on the target device's screen resolution, size, and object hierarchy, the system maintains cross-platform compatibility while preserving the ease of script composition through automated parameter adaptation
3Adaptability or versatility
If image-based technique is used to support comparison and improve design flexibility, then design flexibility is improved, but image recognition consumes a lot of resources
Solution Approach 1:
The system extracts only the essential object information and hierarchical structure from the terminal device interface, rather than performing comprehensive image recognition and comparison. By focusing on extracting key object properties, positions, and relationships, the system achieves design flexibility through object-based testing while significantly reducing resource consumption by avoiding full image processing
4Adaptability or versatility
If object-based technique is used for cross-platform testing, then cross-platform testing is enabled, but compiling of original objects and transformation of program codes is required
Solution Approach 1:
The system creates a copy of the object hierarchy and properties from the terminal device and uses this copied structure to generate the testing script. By copying the object model rather than requiring compilation and code transformation, the system achieves cross-platform testing capability while simplifying the process and avoiding the complexity of recompiling objects or transforming program codes for different platforms
Data Source
AI summary
A testing device and a testing method thereof are provided. The testing device is connected to a terminal device running a graphical user interface (GUI). The testing device runs a testing program to start a recording procedure to execute the following steps: detecting a plurality of actions generated in response to operations on the terminal device; detecting a foreground application of the GUI; reading a plurality of pieces of object information of the foreground application; and determining the actions to record an object property operation of the foreground application and a call command. The testing device further stops the recording procedure to generate and store a script file and a reference log file. The script file includes the object property operation and the call command.