Semantic AI Screen Matching for Faster RPA Workflow Creation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current RPA workflow creation in robotic process automation is time-consuming due to the manual indication of target graphical elements, lacking full automation support.
Innovation Solution
Implementing semantic AI/ML models for automatic semantic matching between source and target screens, providing confidence scores and highlighting matched/unmatched graphical elements, and automatically generating RPA workflow activities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual indication of target graphical elements is used in RPA workflow creation, then developers can precisely specify target elements, but the workflow creation process becomes time-consuming
Solution Approach 1:
The patent replaces the mechanical manual indication process with an AI-based automated system. The AI model automatically identifies and matches target graphical elements between source and target screens, substituting the manual mechanical process of developers indicating elements with automated intelligent recognition, thereby resolving the contradiction between precision and time consumption
Solution Approach 2:
The system enables self-service automation where the AI model autonomously performs the task of identifying and matching graphical elements without requiring developer intervention for each element indication. The workflow creation process serves itself by automatically generating the necessary element mappings, eliminating the time-consuming manual process while maintaining accuracy
2Productivity
If full automated RPA workflow creation is implemented, then workflow creation time is reduced, but the ability to precisely indicate target elements is lost
Solution Approach 1:
The patent introduces an AI model as an intermediary between the automated workflow creation process and the target element identification task. This intermediary performs semantic matching and visual recognition to accurately identify target graphical elements, bridging the gap between automated speed and precise identification accuracy that neither pure automation nor pure manual processes could achieve alone
3Ease of operation
If semantic AI models are used for automatic matching, then manual effort is reduced, but system complexity increases
Solution Approach 1:
The patent extracts the complex semantic matching and visual recognition functionality into a separate, dedicated AI model component. This extraction isolates the complexity within a specialized module that can be independently managed and improved, while the main workflow creation interface remains simple and easy to use, resolving the contradiction between ease of operation and system complexity
Data Source
AI summary
Semantic matching between a source screen or source data and a target screen using semantic artificial intelligence (AI) for robotic process automation (RPA) workflows is disclosed. The source data or source screen and the target screen are selected on a matching interface, semantic matching is performed between the source data/screen and the target screen using an artificial intelligence/machine learning (AI/ML) model, and matching graphical elements and unmatched graphical elements are highlighted, allowing the developer to see which graphical elements match and which do not. The matching interface may also provide a confidence score of the individual matches, provide an overall mapping score, and allow the developer to hide/unhide the matched/unmatched graphical elements. Activities of an RPA workflow may be automatically created based on the semantic mapping that can be executed to perform the automation.