AI-Based UI Element Identification for Stable Software Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web application software tests frequently fail due to minor changes in CSS styling or responsive application layouts, requiring new technologies for effective element identification and tracking to maintain test cases.
Innovation Solution
The technology employs machine learning to identify software application elements by analyzing UI attributes and values, using a combination of generative and discriminative models to produce element definitions and match scores, and incorporates computer vision and natural language processing to improve element identification and tracking across different software versions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional selectors are used to identify UI elements, then test cases can be written with simple syntax, but test cases fail frequently when element attributes or layouts change
Solution Approach 1:
The patent changes the parameters used for element identification from traditional CSS selectors (id, class, tag) to a multi-dimensional parameter set including visual features (color, shape, position), text content, hierarchical context, and attribute combinations. This allows the system to identify elements through multiple alternative parameters, maintaining reliability when one parameter changes due to CSS styling updates or responsive layout adjustments.
Solution Approach 2:
The patent creates a composite identification model that combines multiple types of element descriptors: visual render data, text content, code attributes, and contextual relationships. This composite approach is analogous to composite materials in engineering, where combining different materials creates a system more resilient to individual component failures. Similarly, combining multiple identification parameters makes the test system more resilient to changes in any single attribute.
2Adaptability or versatility
If CSS styling changes or responsive layouts are implemented, then application adaptability improves, but existing test selectors break and require frequent recoding
Solution Approach 1:
The patent performs preliminary actions by capturing and storing multiple alternative descriptors for each UI element during the test setup phase, including visual features, text content, hierarchical context, and attribute combinations. This preliminary documentation of multiple identification pathways ensures that when CSS styling changes or responsive layouts are deployed, the test system already has alternative selectors ready to identify elements without requiring immediate recoding.
Solution Approach 2:
The patent implements a feedback mechanism where the system continuously monitors element identification success rates and automatically adjusts selector strategies. When an element cannot be identified using traditional selectors due to CSS changes, the system feedback loop triggers alternative identification methods using visual features, text content, or contextual relationships, thereby adapting to responsive layout changes without manual intervention.
3Measurement precision
If multiple element attributes are monitored for identification, then element identification accuracy improves, but processing complexity and computational resources increase
Solution Approach 1:
The patent applies local quality by assigning different levels of importance and monitoring intensity to different element attributes based on their stability and discriminative power. Critical attributes such as text content and hierarchical context are monitored with high precision, while less stable attributes like CSS styles are monitored with lower intensity. This localized approach to attribute monitoring maintains high identification accuracy while reducing overall processing complexity by focusing computational resources on the most reliable indicators.
Data Source
AI summary
Provided herein is technology relating to selecting an element in a software application and particularly, but not exclusively, to systems and methods for identifying a target element for testing a software application using artificial intelligence.