Multidimensional GUI Element Indexing for Automated Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for locating graphical user interface (GUI) elements during automated testing are inefficient, particularly in multilingual environments where layout changes and resolution inconsistencies occur, leading to inconsistent element mapping between different language screens.
Innovation Solution
The method employs a machine learning model to index GUI elements using a four-dimensional indexing system, which includes left-to-right and top-to-bottom spatial dimensions, hierarchical grouping, and a time dimension that updates with UI changes, allowing for dynamic and consistent element location across different language versions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional methods are used to locate GUI elements, then the testing process is simple to implement, but the element mapping becomes inconsistent across different language screens due to layout changes
Solution Approach 1:
The patent introduces a fourth dimension (time) to the traditional two-dimensional spatial indexing system. By adding the time dimension, the system can track GUI element positions across different language versions and layout configurations, enabling consistent element mapping while accommodating layout changes through temporal tracking of element positions.
Solution Approach 2:
The patent implements a dynamic indexing system that automatically updates element positions and mappings when layout changes occur. The system continuously monitors and records position changes across different language screens, allowing the indexing to adapt dynamically rather than remaining static, thus maintaining reliability despite layout variations.
2Measurement precision
If automated testing is performed without multidimensional indexing, then the testing speed is fast, but the testing accuracy decreases due to layout changes and resolution inconsistencies
Solution Approach 1:
The patent performs preliminary indexing of GUI elements into a multidimensional index structure before actual testing begins. By pre-establishing the comprehensive index with spatial and temporal dimensions, the system prepares accurate element location data in advance, enabling both high precision during testing and maintaining productivity through efficient retrieval from the pre-built index.
Solution Approach 2:
The patent replaces traditional mechanical or rule-based element location methods with a machine learning-based multidimensional indexing system. This substitution enables the system to intelligently handle layout changes and resolution inconsistencies, achieving both high measurement precision and maintained productivity through automated intelligent indexing rather than rigid mechanical rules.
3Adaptability or versatility
If multidimensional indexing with machine learning is implemented, then element mapping consistency across languages improves, but the system complexity and resource requirements increase
Solution Approach 1:
The patent creates a universal multidimensional indexing system that serves multiple functions: it indexes elements across different language versions, handles layout changes, accommodates resolution inconsistencies, and maintains element mapping consistency. By making the indexing system multi-functional rather than language-specific, it achieves broad adaptability while managing complexity through a single unified approach that works across all scenarios.
Data Source
AI summary
Disclosed are techniques for automated locating of user interface elements during graphical user interface testing. When a graphical user interface (GUI) is received for testing, images of the GUI are inputted to a machine learning algorithm, where image processing techniques are applied to identify groups of user interface elements and their constituent elements. Multi-dimensional index values are assigned to groups and elements corresponding to their positions within the GUI. Automated testing of the user interface elements of the GUI is performed by locating the user interface elements by their index values. If an element is not found, a scrolling technique is applied to generate an expanded virtual GUI of one or more groups of user interface elements, and the machine learning algorithm refreshes the index values using the expanded virtual GUI.