Automated Testing Using SIFT Image Patches for Cross-Platform UI Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated testing systems face challenges in accurately identifying graphical elements on different screen sizes and resolutions, leading to unreliable input operations during testing of software applications, especially when the testing system is trained on one platform but executed on another with varying display parameters.
Innovation Solution
The use of a scale-invariant feature transform (SIFT) algorithm to detect feature points in image patches, allowing for consistent input operations across different screen sizes and resolutions by automatically adjusting the image patch boundaries to capture sufficient feature points, thereby ensuring accurate identification of graphical elements during automated testing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If the testing system uses machine vision to identify graphical elements on the AUT, then the system can perform automated testing without human intervention, but the processing demands increase significantly when selecting large portions of the screen for analysis
Solution Approach 1:
The patent divides the screen into multiple image patches instead of analyzing the entire screen. Each image patch corresponds to a specific graphical element or region of interest. This segmentation reduces the processing area from the full screen to smaller, manageable patches, significantly lowering computational demands while maintaining automated testing capability.
Solution Approach 2:
The patent applies different processing qualities to different regions of the screen. High-resolution analysis is applied only to image patches containing graphical elements that need testing, while other areas receive minimal or no processing. This local quality approach optimizes resource allocation by focusing computational power where it is most needed.
2Use of energy by moving object
If the testing system selects a small portion of the display for analysis, then processing demands are reduced, but the target graphical element may not be found, especially when screen sizes differ from the training system
Solution Approach 1:
The patent implements dynamic image patch adjustment where the boundaries and sizes of image patches are not fixed but adapt based on the detected graphical elements and screen characteristics. The system dynamically resizes and repositions image patches to ensure they capture the relevant graphical elements regardless of screen size variations, maintaining recognition accuracy across different platforms.
Solution Approach 2:
The patent changes key parameters of the image patches including size, position, and resolution based on the specific testing context and screen characteristics. By adjusting these parameters dynamically, the system ensures that image patches contain sufficient visual information for reliable graphical element identification while adapting to different screen sizes and resolutions.
3Ease of manufacture
If the testing system uses coordinate-based input operations, then simple automated testing can be performed, but the system fails when screen resolution or size differs from the training platform
Solution Approach 1:
The patent replaces the mechanical coordinate-based input system with an image-based recognition system. Instead of specifying fixed coordinates for input operations, the system uses image patches to visually identify graphical elements and automatically determine the appropriate input targets. This substitution makes the testing system adaptable to different screen resolutions and sizes while maintaining cross-platform compatibility.
4Reliability
If the system analyzes the entire screen for graphical elements, then comprehensive testing coverage is achieved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the screen into multiple image patches, each focusing on a specific graphical element or functional area. This segmentation allows the system to analyze only the relevant portions of the screen rather than the entire display, significantly reducing processing time while maintaining comprehensive testing coverage across all critical graphical elements.
Solution Approach 2:
The patent applies partial action by analyzing only the necessary image patches containing graphical elements rather than the entire screen. This selective approach provides sufficient testing coverage for the functional elements under test without the excessive processing overhead of full-screen analysis, achieving an optimal balance between coverage and efficiency.
Data Source
AI summary
Applications under test (AUT) may be tested by automated testing systems utilizing machine vision to recognize visual elements presented by the AUT and apply inputs to graphical elements, just as a human would. By utilizing the smallest image patch available, processing demands of the testing system are minimized. However, the image patch used to identify a portion of an AUT must be identifiable to the automated system. By selecting image patches that comprise the smallest size, but can be identified in an AUT by an automated system using machine vision, even as the AUT display is resized, reproportioned, noisy, or otherwise altered from the testing platform that was utilized for training.