RPA Script Fallback Identification Using Image and Text Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Robotic process automation (RPA) systems face challenges in unambiguously identifying target user interface elements due to changes in the appearance of interfaces, such as positioning, color schemes, and fonts, between design and runtime, leading to potential failures in automation tasks.

Innovation Solution

The method involves generating an RPA script that encodes a target ID, image, and text of the target element, allowing the software robot to search for matching elements at runtime and identify them based on these encodings, even if the target ID does not match, ensuring robustness and scalability by using image and text data as fallbacks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If RPA systems rely on target ID encoding for element identification, then the identification process is simple and fast, but the system fails when interface appearance changes between design and runtime

Engineering Contradiction:

ImproveRPA execution reliabilityVSAvoidInterface change adaptability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system performs preliminary actions by capturing and storing multiple representations (screenshot images and text transcriptions) of target UI elements during the design phase. These pre-captured data are stored in the RPA script alongside the target ID, enabling the system to have fallback identification methods ready before runtime interface changes occur.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates copies of the target element's visual appearance (screenshot) and text content, storing these copies in the RPA script. At runtime, if the target ID cannot be matched, the system uses these pre-stored copies to identify the element through image matching or text comparison, thus adapting to interface changes.

Inventive Principle:

Principle #26Copying

2Adaptability or versatility

If RPA systems use multiple identification methods (image and text) as fallbacks, then robustness against interface changes improves, but the script complexity and data processing requirements increase

Engineering Contradiction:

ImproveInterface change adaptabilityVSAvoidRPA script complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The RPA script is designed with multi-functionality, incorporating three identification methods (target ID, image matching, and text transcription) within a single script structure. This universal approach allows the same script to handle both stable interfaces (using simple ID matching) and changed interfaces (using image or text fallbacks) without requiring separate scripts.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system applies different identification methods with varying degrees of complexity based on local conditions. The target ID method is used as the primary simple approach, while image and text methods are prepared as localized fallback options that are only activated when needed, thus managing overall script complexity through conditional complexity.

Inventive Principle:

Principle #3Local quality

3Measurement precision

If RPA systems capture and store image data of target elements, then identification accuracy under interface changes improves, but the data size and processing time increase

Engineering Contradiction:

ImproveElement identification accuracyVSAvoidRuntime processing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system implements partial action by using image matching only when necessary (when target ID matching fails). The primary identification uses simple target ID comparison, and image-based identification is applied partially as a fallback mechanism, thus avoiding the full processing overhead of image analysis in all cases while maintaining high accuracy when needed.

Inventive Principle:

Principle #16Partial or excessive action

4Reliability

If RPA systems implement fallback mechanisms for target identification, then execution success rate improves, but the identification process becomes more complex and slower

Engineering Contradiction:

ImproveRPA execution reliabilityVSAvoidElement identification speed

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The system performs preliminary action by pre-capturing and storing image screenshots and text transcriptions during the design phase. This preparation work is done in advance, so that at runtime, the fallback mechanisms can quickly compare against pre-processed data rather than performing heavy image analysis from scratch, thus maintaining speed while ensuring reliability.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11556313B2Providing image and text data for automatic target selection in robotic process automation

Publication Date: 2023.01.17 UIPATH INC

AI summary

A software robot is designed to carry out an activity (e.g., a mouse click, a text input, etc.) on a target element (e.g., a button, an input field, etc.) of a user interface. The robot's code specification is configured to include an on-screen image of the target element and a text displayed by the target element. The robot is configured to automatically identify the target element at runtime according to an element ID specified in the source-code of the user interface, and when such identification fails, to identify the target element according to the text and image stored in the robot's code.