Semantic UI Target Matching for Robust RPA Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robotic process automation (RPA) systems struggle to unambiguously identify target elements in user interfaces due to changes in UI design, such as element movement, renaming, resizing, or color scheme changes, leading to failed target identification.
Innovation Solution
A computer system identifies runtime instances of target elements in user interfaces by comparing the meaning of design-time target labels with runtime labels using semantic similarity, allowing RPA activities to execute on the correct elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional UI element identification methods (based on position, color, label) are used, then the RPA robot can identify target elements during design-time, but the identification fails when UI design changes occur at runtime
Solution Approach 1:
The patent transforms the target identification approach by changing from relying on fixed UI parameters (position, color, label text) to using semantic meaning extraction and comparison. The system extracts semantic representations of UI elements that remain consistent despite visual or positional changes, allowing reliable identification across UI variations.
Solution Approach 2:
The patent introduces semantic meaning as an intermediary layer between the RPA robot and the UI elements. Instead of directly comparing visual characteristics, the system uses semantic extraction and comparison as a mediator to identify target elements, making the identification process robust to UI design changes.
2Measurement precision
If the RPA robot uses fixed programmatic characteristics (attribute-value pairs) to identify UI elements, then identification is precise for the design-time UI, but it becomes fragile when UI elements are renamed or repositioned
Solution Approach 1:
The patent changes the identification parameters from fixed attribute-value pairs (position coordinates, color codes, exact label text) to semantic representations that capture the meaning and function of UI elements. This allows the system to maintain precise identification even when visual attributes change.
Solution Approach 2:
The patent makes the identification system dynamic by allowing flexible matching of semantic meanings rather than requiring exact matches of fixed attributes. The system can adapt to renamed or repositioned elements by comparing their semantic representations, which reflect their functional meaning rather than static properties.
3Adaptability or versatility
If the UI is developed and maintained independently of the RPA system, then UI development flexibility is improved, but the RPA robot cannot track changes and fails to identify updated elements
Solution Approach 1:
The patent uses semantic meaning extraction and comparison as an intermediary mechanism that bridges the independent UI development process and the RPA automation system. This mediator allows the RPA robot to identify UI elements based on their functional meaning rather than their implementation details, maintaining automation continuity despite independent UI changes.
Solution Approach 2:
The patent creates a universal identification approach that works across different UI designs and implementations. By focusing on semantic meaning rather than specific UI characteristics, the system becomes universally applicable to various UI configurations developed independently, maintaining automation reliability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Some embodiments automatically identify a target of a robotic process automation (RPA) activity (e.g., a button to click, an input field to fill out) according to a semantic similarity between a design-time label of the target and a label of a target candidate selected from a runtime instance of the target UI. Semantic similarity herein denotes likeness of meaning, as opposed to wording. Some embodiments employ a language model (LM) to quantify semantic similarity.