LLM Interface Automation Using Screen Representations and Action Blocks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Creating software automations for multipart computing systems and intricate software is complex, requiring advanced expertise, and conventional automations are slow, resource-intensive, and prone to errors, especially when interacting with APIs or web applications.

Innovation Solution

An automation engine using generative pretrained transforms (GPTs) and large language models (LLMs) generates workflows and automates interfaces by extracting DOMs from web pages, determining next steps, and configuring API activities based on user queries, improving processing speed and resource utilization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If conventional software automation methods are used, then automation can be achieved, but the process requires advanced expertise and is complex to implement

Engineering Contradiction:

Improveautomation capabilityVSAvoidimplementation complexity

Core Design Contradiction:

Extent of automationVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary system that includes a screen reader component and automation rule generator that acts as a mediator between the user interface and the automation execution engine. This intermediary captures UI elements through screen reading, generates automation rules from captured data, and translates user actions into executable automation commands, thereby reducing implementation complexity while maintaining automation capability

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system enables self-service automation by automatically capturing UI elements through screen reading, generating automation rules without requiring manual programming, and executing automations based on user-defined criteria. The system serves itself by autonomously identifying UI elements, creating automation logic, and managing execution, eliminating the need for advanced expertise in automation programming

Inventive Principle:

Principle #25Self-service

2Productivity

If conventional software automations are used, then tasks can be automated, but they are slow and cumbersome to operate

Engineering Contradiction:

Improveautomation executionVSAvoidoperation speed

Core Design Contradiction:

ProductivityVSSpeed

Solution Approach 1:

The system performs preliminary actions by pre-capturing UI element data through screen reading, pre-generating automation rules from captured data, and pre-configuring automation parameters before execution. This preliminary preparation enables faster execution during actual automation operations, as the system does not need to analyze or interpret UI elements in real-time during execution

Inventive Principle:

Principle #10Preliminary action

3Extent of automation

If conventional software automations are used, then automation functionality can be provided, but they consume excessive processing power

Engineering Contradiction:

Improveautomation functionalityVSAvoidprocessing power consumption

Core Design Contradiction:

Extent of automationVSUse of energy by moving object

Solution Approach 1:

The system extracts and separates the screen reading function from the automation execution function. The screen reader component captures UI element data independently and stores it for later use, while the automation engine executes tasks using this pre-captured data. This extraction eliminates redundant processing during automation execution, reducing processing power consumption while maintaining full automation functionality

Inventive Principle:

Principle #2Taking out (Extraction)

4Extent of automation

If conventional software automations are used, then automation can be implemented, but they overutilize memory resources

Engineering Contradiction:

Improveautomation implementationVSAvoidmemory utilization

Core Design Contradiction:

Extent of automationVSQuantity of substance

Solution Approach 1:

The system segments memory usage by separating screen reading data storage from automation execution memory requirements. UI element data captured by the screen reader is stored in dedicated memory structures, while the automation engine uses separate memory resources for execution. This segmentation optimizes memory allocation and prevents overutilization by ensuring each component uses only the memory necessary for its specific function

Inventive Principle:

Principle #1Segmentation

5Productivity

If conventional software automations are used, then automation processes can run, but they are prone to processor errors that lock computing resources

Engineering Contradiction:

Improveautomation process executionVSAvoidprocessor error rate

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements beforehand cushioning by incorporating error handling and recovery mechanisms in the automation rule execution engine. The engine monitors execution processes, detects potential errors before they cause system locks, and implements recovery actions such as rolling back to previous stable states or alternative execution paths. This protective measure ensures continuous reliable operation while maintaining productivity

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS20260079724A1Automating interfaces using a combination of representations and generative pretrained transforms

Publication Date: 2026.03.19 UIPATH INC

AI summary

A method is provided. The method is executed by an automation engine implemented as a computer program within a computing environment. The method comprising includes outputting a combined screen representation to a subsequent user interface comprising demonstrations. The method comprising includes generating a step with blocks by using a large language model to match the step with action block descriptions. The method comprising includes automating the interface based on the action blocks.