Document Extraction ML for RPA UI Element Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Designing robotic process automation (RPA) robots is time-consuming and requires numerous manual tasks due to the need for users to manually declare application screens and user interface elements, which is tedious, especially for complex applications.

Innovation Solution

Utilizing a document information extraction machine learning model to automatically identify user interface elements from screenshots, reducing the manual effort by presenting a list of identified areas and elements for selection and declaration in the RPA design process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If manual declaration of UI elements is used in RPA robot design, then the robot can be configured to perform automated tasks, but the design process becomes time-consuming and tedious

Engineering Contradiction:
Improveautomation of RPA robot designVSAvoiddesign time
Core Design Contradiction:
Extent of automationVSLoss of time

Solution Approach 1:

The system enables self-service by allowing the RPA design system to automatically capture and declare UI elements without requiring manual user intervention. The screenshot capture component automatically captures UI elements, and the declaration component automatically declares them as RPA entities, making the system serve itself rather than requiring continuous human input.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The system performs preliminary action by capturing screenshots and declaring UI elements before the actual RPA robot execution. The capture phase and declaration phase are completed in advance, creating a ready-to-execute automation script that can then run without further manual intervention.

Inventive Principle:
Principle #10Preliminary action

2Productivity

If manual tasks are performed to declare application screens and UI elements, then the RPA robot can be designed, but numerous manual tasks are required

Engineering Contradiction:
ImproveRPA robot design efficiencyVSAvoidease of RPA robot design
Core Design Contradiction:
ProductivityVSEase of operation

Solution Approach 1:

The system enables self-service by allowing the RPA design system to automatically capture and declare UI elements without requiring manual user intervention. The screenshot capture component automatically captures UI elements, and the declaration component automatically declares them as RPA entities, making the system serve itself rather than requiring continuous human input.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The system replaces manual mechanical operations with automated computational processes. Instead of manually clicking and declaring each UI element, the system uses screenshot capture technology and automated declaration components to identify and declare elements programmatically, substituting human manual labor with automated systems.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

3Loss of time

If automated screenshot capture and element declaration is implemented, then design time is reduced, but the system complexity increases

Engineering Contradiction:
Improvedesign timeVSAvoidsystem complexity
Core Design Contradiction:
Loss of timeVSDevice complexity

Solution Approach 1:

The system applies segmentation by dividing the RPA design process into distinct functional components: a screenshot capture component, a declaration component, and an execution component. Each component has a specific responsibility, making the overall complex system manageable through modular organization where each segment can be independently developed and maintained.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The system achieves universality by creating a multi-functional RPA design system that can handle multiple types of UI elements across different applications through a single unified framework. The screenshot capture and declaration components are designed to work with various UI types (buttons, text fields, menus, etc.) without requiring application-specific customization, making the system universally applicable.

Inventive Principle:
Principle #6Universality (Multi-functionality)

Data Source

PatentUS12430939B2Machine learning integration in robotic process automation
Publication Date: 2025.09.30 SAP SE
  • US12430939B2 patent drawing
  • US12430939B2 patent drawing
  • US12430939B2 patent drawing

AI summary

Automation of robotic process automation design may be improved by integrating machine learning. A screenshot of a user interface of an application to be automated is captured. Then a selection of an area of the screenshot is determined. The area is provided as input to a document information extraction machine learning model configured to output labels and coordinates for user interface elements in the area. Labels for respective user interface elements of the one or more user interface elements and a respective set of coordinates are identified for each of the user interface elements. The user interface elements and respective criteria are declared as robotic process automation entities in a robotic process automation object model. Actions are assigned to the robotic process automation elements. And then the actions are performed on the application to be automated based on the respective criteria for the robotic process automation elements.