LLM UI Automation Using Cached Actions to Reduce Processing Time

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies face challenges in efficiently automating user interface (UI) operations using large language models (LLMs) due to the complexity of mapping natural language instructions to precise UI actions, particularly in software applications, leading to inefficiencies and resource wastage.

Innovation Solution

A system utilizing an LLM-based UI assistant that includes an autonomous agent leveraging application and UI platform documentation, parameter and prompt caching, and feedback mechanisms to generate and execute UI actions accurately and efficiently, optimizing the mapping process through parameterization and caching strategies.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If LLMs are used to generate UI actions from natural language instructions, then automation capability is improved, but computational resource consumption increases

Engineering Contradiction:

Improveautomation capabilityVSAvoidcomputational resource consumption

Core Design Contradiction:

Extent of automationVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary actions by caching frequently accessed UI element information, parameter mappings, and prompt responses before they are actually needed for automation tasks. This pre-computation and storage of common patterns reduces the computational burden during real-time UI automation operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates copies of successful UI action patterns and stores them in a cache database. When similar tasks arise, instead of generating new actions from scratch using computationally intensive LLMs, the system retrieves and reuses cached action patterns, significantly reducing computational resource consumption while maintaining automation capability.

Inventive Principle:

Principle #26Copying

2Manufacturing precision

If LLMs process every natural language instruction to generate UI actions, then task accuracy is improved, but processing time increases

Engineering Contradiction:

Improvetask accuracyVSAvoidprocessing time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The system performs preliminary analysis of natural language instructions to identify patterns and extract key parameters before invoking the LLM. By pre-processing and categorizing instructions, the system can make informed decisions about whether to use cached responses or generate new actions, reducing unnecessary LLM processing time while maintaining accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system copies and retrieves pre-computed UI action sequences from the cache for common and repetitive tasks. This allows the system to maintain high task accuracy by using proven action patterns while dramatically reducing processing time compared to generating new actions for each task.

Inventive Principle:

Principle #26Copying

3Adaptability or versatility

If the system generates unique UI actions for every instruction, then adaptability is improved, but system complexity increases

Engineering Contradiction:

ImproveadaptabilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments the UI automation task into distinct components: natural language processing, parameter extraction, action pattern matching, and execution. By dividing the complex task into manageable segments, the system can maintain adaptability to various instructions while reducing overall system complexity through modular architecture.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system uses cached action patterns as templates that can be adapted to similar new tasks. Rather than generating entirely unique actions for every instruction, the system retrieves relevant cached patterns and adapts them to the current context, maintaining versatility while simplifying the generation process.

Inventive Principle:

Principle #26Copying

4Loss of energy

If parameter caching is implemented to reduce computational resources, then resource efficiency is improved, but data management complexity increases

Engineering Contradiction:

Improveresource efficiencyVSAvoiddata management complexity

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The system extracts and separates parameter data from the main LLM processing flow, storing frequently accessed parameters in a dedicated cache database. This extraction allows the LLM to focus on high-level decision-making while cached parameters handle routine data retrieval, improving resource efficiency with manageable data management through structured storage and retrieval mechanisms.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentEP4610808A1Using large language model agents for robust and performant user interface automation

Publication Date: 2025.09.03 WORKDAY INC

Figure 1
Figure 2
Figure 3

AI summary

In some implementations, the techniques described herein relate to a method including: receiving, by a processor, a natural language instruction from a client device, the natural language instruction describing a task utilizing a software application; generating, by the processor, a user interface action representing the natural language instruction, the user interface action generated by a large language model responsive to an input prompt; executing, by the processor, the user interface action within the software application; and transmitting, by the processor, a result of executing the user interface action to the client device.