Visual Analysis for Adaptive Mobile Device Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automated testing techniques for mobile devices struggle with adaptability, failing when changes are made to device elements such as icon names and locations, requiring time-consuming and expensive manual testing across multiple device types.
Innovation Solution
A machine learning model trained on interaction data from testers to identify visual elements and simulate user actions, allowing automated testing to adapt to changes in mobile devices and operating systems without requiring testing scripts or code-level changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional automated testing techniques are used, then testing can be automated, but the system lacks adaptability when device elements change
Solution Approach 1:
The patent replaces traditional script-based mechanical testing control with machine vision and optical recognition systems. The testing system captures screenshots and uses image processing to dynamically identify UI elements, replacing rigid predefined scripts with adaptive visual analysis that can handle device changes automatically.
Solution Approach 2:
The system dynamically changes testing parameters based on visual analysis of device screenshots. Instead of fixed test scripts, the system adjusts element locations, identifiers, and interaction parameters in real-time based on what it visually detects, enabling adaptability while maintaining automation.
2Adaptability or versatility
If manual testing is performed across multiple device types, then adaptability is maintained, but testing time and costs increase
Solution Approach 1:
The patent creates a universal testing system that can operate across multiple device types and operating systems using a single automated platform. The machine vision approach is device-agnostic, allowing one system to test various smartphones regardless of brand or OS, eliminating the need for separate manual testing teams for each device type.
Solution Approach 2:
The system uses screenshot copying to create visual representations of device screens, then analyzes these copies to determine testing actions. This allows the system to understand and test different device interfaces without physical interaction, enabling rapid cross-device testing.
3Reliability
If testing scripts are updated to accommodate device changes, then testing accuracy is maintained, but maintenance complexity increases
Solution Approach 1:
The testing system performs self-updating through automatic visual analysis. When device interfaces change, the system independently detects new element locations and characteristics through screenshot analysis, automatically adjusting its testing approach without requiring external script updates or manual reconfiguration.
Solution Approach 2:
The system continuously monitors device screenshots during testing and uses this visual feedback to adjust its testing actions in real-time. If elements are found at different locations or with different characteristics than expected, the system adapts its behavior based on this feedback, maintaining accuracy without complex update procedures.
Data Source
AI summary
Described herein are techniques that may be used to automate testing of services on mobile devices using visual analysis. In some embodiments, a machine learning model is trained using interaction data received from a number of mobile devices by correlating user selections with visual elements (e.g., icons). During execution of a testing routine on a mobile device, screenshots are obtained of a screen of the mobile device and provided to the machine learning model. An action is generated based on the provided screenshot that simulates a user action (e.g., a user touch on the screen of the mobile device) at a location of an icon or other visual element associated with the testing routine. These steps are repeated until an end-state of the testing routine is detected.