LLM Interface Automation Using Screen and DOM Representations

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Creating software automations for multipart computing systems and intricate software is complex, requiring advanced technical expertise, and conventional automations are slow, resource-intensive, and prone to errors, especially when interacting with APIs or web applications.

Innovation Solution

An automation engine utilizing generative pretrained transforms (GPTs) and large language models (LLMs) generates automations on web applications, extracts DOMs efficiently, and configures workflows based on user queries, improving processor efficiency and reducing errors.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional software automations are used to interact with APIs or web applications, then automation capability is provided, but the system becomes slow, resource-intensive, and prone to errors

Engineering Contradiction:

Improveautomation accuracyVSAvoidautomation speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent replaces conventional mechanical automation approaches (script-based, rule-based systems) with an AI-driven neural network system that processes screenshots and interface elements to generate automations, thereby improving both reliability and speed simultaneously

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system dynamically adjusts automation parameters by using the neural network to analyze interface states and generate optimized automation sequences, changing from static pre-programmed actions to dynamic adaptive actions based on real-time interface analysis

Inventive Principle:

Principle #35Parameter changes

2Reliability

If senior software engineers manually create and configure computer actions, then automation quality improves, but the complexity and time required increases significantly

Engineering Contradiction:

Improveautomation qualityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system enables self-service automation generation where the neural network automatically analyzes interface screenshots, identifies actionable elements, and generates automation code without requiring manual programming expertise from users

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent performs preliminary analysis of interface screenshots and element hierarchies before generating automations, pre-processing the visual information to extract meaningful structural data that simplifies subsequent automation generation

Inventive Principle:

Principle #10Preliminary action

3Extent of automation

If conventional automation methods are used, then basic automation functions are achieved, but excessive processing power and memory are consumed

Engineering Contradiction:

Improveautomation capabilityVSAvoidprocessing power consumption

Core Design Contradiction:

Extent of automationVSUse of energy by moving object

Solution Approach 1:

The system extracts only the essential interface elements and structural information from screenshots using the neural network, rather than processing entire interface renderings, thereby reducing computational overhead while maintaining automation capability

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent segments the interface analysis into distinct processing stages (screenshot capture, element identification, hierarchy construction, automation generation), allowing efficient resource allocation to each stage and avoiding unnecessary processing

Inventive Principle:

Principle #1Segmentation

Data Source

PatentEP4711922A1Automating interfaces using a combination of representations and generative pretrained transforms

Publication Date: 2026.03.18 UIPATH INC

Figure 1
Figure 2
Figure 3

AI summary

A method is provided. The method is executed by an automation engine implemented as a computer program within a computing environment. The method comprising includes outputting a combined screen representation to a subsequent user interface comprising demonstrations. The method comprising includes generating a step with blocks by using a large language model to match the step with action block descriptions. The method comprising includes automating the interface based on the action blocks.