Machine Learning Object Identification for GUI Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning-based functional testing for graphical user interfaces faces performance issues when capturing images and identifying graphical objects, leading to slow testing times due to repeated machine learning processes and lengthy screenshot captures, especially on devices like smartphones.
Innovation Solution
A process is introduced to record and update machine learned software objects with the location information of graphical objects, allowing for direct identification and interaction without repeated machine learning calls, using AIUtil.BeginSceneBlock and AIUtil.EndSceneBlock to capture and reuse machine learned information, thereby reducing testing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning processes are used to identify graphical objects in each screenshot, then identification accuracy is improved, but testing time increases significantly
Solution Approach 1:
The patent applies preliminary action by capturing graphical object location information once during application installation and storing it in a database. This pre-captured data is then reused during testing operations, eliminating the need to repeatedly execute machine learning processes for the same objects, thus reducing testing time while maintaining identification accuracy.
Solution Approach 2:
The patent uses copying by creating and storing copies of graphical object information (location, coordinates, properties) in a database during installation. These copied data representations are then referenced during testing instead of重新 capturing them through machine learning, significantly reducing processing time while preserving the necessary identification information.
2Reliability
If machine learning processes are executed repeatedly for each testing operation, then object identification reliability is improved, but productivity decreases
Solution Approach 1:
The system performs preliminary capture of graphical object information during application installation and stores it for future use. This preliminary action ensures reliable object identification data is available without needing to repeat machine learning processes during each testing operation, thereby improving productivity while maintaining reliability.
Solution Approach 2:
The patent applies local quality by differentiating between installation-time operations (where comprehensive machine learning analysis is performed) and testing-time operations (where pre-captured local object data is reused). This approach maintains high reliability where needed while improving overall productivity by avoiding redundant processing.
3Measurement precision
If comprehensive machine learning analysis is performed on each screenshot, then testing accuracy is improved, but device resource consumption increases
Solution Approach 1:
The patent performs comprehensive machine learning analysis preliminarily during application installation and stores the results. During subsequent testing operations, the system uses the pre-analyzed data without repeating the energy-intensive machine learning processes, thereby maintaining testing accuracy while significantly reducing device energy consumption.
Solution Approach 2:
The system creates copies of the machine learning analysis results and stores them in a database. These copied data representations are then used during testing instead of重新 performing the energy-intensive analysis, preserving testing accuracy while reducing device resource consumption.
Data Source
AI summary
An image of a graphical user interface is captured. A process is started to record identification of a graphical object (e.g., a text field) in the image of the graphical user interface. Recording identification of the graphical object in the image of the graphical user interface is based on machine learning. An end to the process to record identification of the graphical object in the image of the graphical user interface is identified. A machine learned software object is updated. For example, the machine learned software object is updated with a location of the graphical object. The updated machine learned software object then is used to test the graphical object. For example, to automatically fill in the text field using a test script.