Automated Mobile Testing Through Visual Behavioral Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automated testing techniques for mobile devices lack adaptability and fail when changes are made to device elements such as icon names and locations, requiring manual reconfiguration and limiting testing across different device models and operating systems.
Innovation Solution
A machine learning model trained on user interaction data correlates visual elements with actions to dynamically predict and execute testing routines on various mobile devices and operating systems, adapting to changes without predefined scripts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional automated testing techniques are used, then testing can be performed on mobile devices, but the testing lacks adaptability and fails when changes are made to device elements such as icon names and locations
Solution Approach 1:
The system uses machine learning models that automatically learn from user interaction data and adapt to device changes without manual reconfiguration. The behavioral model dynamically determines which objects to select based on learned patterns, enabling the testing system to self-adjust when device elements change.
Solution Approach 2:
The system transitions from static, predefined testing scripts to dynamic, data-driven testing parameters. Machine learning models process training data to generate adaptive testing sequences that can respond to changes in device interface parameters such as icon locations and names.
2Reliability
If manual reconfiguration is performed to adapt testing to device changes, then testing reliability can be maintained, but the complexity and time required for testing increases
Solution Approach 1:
The patent replaces manual reconfiguration processes with automated machine learning-based decision-making. The behavioral model substitutes human analysts by dynamically determining which objects to interact with based on learned user behavior patterns, eliminating the need for manual script updates.
Solution Approach 2:
Machine learning models serve as intermediaries between the testing system and the mobile device interface. These models process screenshots and training data to bridge the gap between static testing scripts and dynamic device interfaces, automatically adapting to changes without direct human intervention.
3Extent of automation
If predefined testing scripts are used, then testing can be automated, but the testing cannot adapt to different device models and operating systems
Solution Approach 1:
The machine learning models are trained on diverse user interaction data from multiple device models and operating systems, enabling a single automated testing system to universally adapt to various devices. The behavioral model learns generalizable patterns that transfer across different platforms and interfaces.
Solution Approach 2:
The testing system transitions from static predefined scripts to dynamic adaptive sequences. The machine learning models continuously process screenshots and determine next actions based on current device state, enabling real-time adaptation to different device models and operating systems during execution.
Data Source
AI summary
Described herein are techniques that may be used to automate testing of services on mobile devices using visual analysis. In some embodiments, a behavioral model or other machine learning model is trained using training data collected while testers use mobile devices to test the services. During execution of a testing routine on a mobile device, screenshots are obtained of a screen of the mobile device and provided to the machine learning model. The behavioral model or other machine learning model can use the provided screenshot to determine an action that simulates a user action (e.g., a user touch on the screen of the mobile device) at a location of an icon or other visual element associated with the testing routine. These steps are repeated until an end-state of the testing routine is detected.