Semantic UI Target Identification for Robust RPA Automation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing robotic process automation (RPA) systems struggle to unambiguously identify target user interface elements due to changes in the design and appearance of UIs, leading to failures in automated interactions.

Innovation Solution

The system identifies target UI elements by comparing the meaning of design-time target labels with runtime labels using semantic similarity, allowing RPA activities to execute on the correct elements despite UI changes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If programmatic and visual characteristics are used to identify target UI elements, then target identification precision is improved, but reliability deteriorates when UI design changes occur

Engineering Contradiction:

Improvetarget identification precisionVSAvoidautomation reliability

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent transforms the identification parameters from static programmatic attributes and visual characteristics to dynamic semantic meanings. By using natural language processing to extract and compare semantic meanings of UI element labels, the system adapts to UI design changes while maintaining reliable identification. The semantic meaning remains consistent even when programmatic attributes or visual characteristics change.

Inventive Principle:

Principle #35Parameter changes

2Manufacturing precision

If exact label matching is used for target identification, then manufacturing precision is improved, but adaptability deteriorates when UI labels change

Engineering Contradiction:

Improvetarget identification accuracyVSAvoidUI change adaptability

Core Design Contradiction:

Manufacturing precisionVSAdaptability or versatility

Solution Approach 1:

The patent introduces semantic meaning as an intermediary layer between the UI label and the identification process. Instead of directly matching labels, the system converts labels to semantic meanings using natural language processing. This intermediary allows flexible interpretation of labels, enabling accurate identification even when labels are renamed or rephrased in the updated UI.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If semantic similarity comparison is used for target identification, then adaptability to UI changes is improved, but device complexity increases

Engineering Contradiction:

ImproveUI variation adaptabilityVSAvoididentification system complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent replaces traditional mechanical matching mechanisms (exact string comparison of labels) with intelligent processing based on natural language semantics. By using pre-trained language models to compute semantic similarities, the system achieves robust adaptability to UI changes. The complexity is managed by leveraging existing NLP infrastructure rather than building custom semantic analysis systems.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20250370773A1Semantic Target Identification for User Interface (UI) Automation

Publication Date: 2025.12.04 UIPATH INC

AI summary

Some embodiments automatically identify a target of a robotic process automation (RPA) activity (e.g., a button to click, an input field to fill out) according to a semantic similarity between a design-time label of the target and a label of a target candidate selected from a runtime instance of the target UI. Semantic similarity herein denotes likeness of meaning, as opposed to wording. Some embodiments employ a language model (LM) to quantify semantic similarity.