GUI Widgets for LLM-Prompted Application Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Digital adoption platforms (DAPs) lack adequate text analysis and generation capabilities, limiting their ability to provide comprehensive assistance to users in performing digital tasks, particularly in analyzing and generating text-based content.
Innovation Solution
Integrate Large Language Models (LLMs) with DAP building blocks to enhance the assistance layer, enabling widgets to perform text analysis and generation tasks, while ensuring reliability and privacy through restricted use and data anonymization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Large Language Models are integrated with DAP building blocks to enhance text analysis and generation capabilities, then the assistance quality and context-awareness are improved, but the system complexity and computational resource requirements increase
Solution Approach 1:
The system segments the DAP architecture by introducing distinct widget components that can independently integrate LLM capabilities. Each widget is a self-contained unit that can perform text analysis or generation tasks, allowing the system to scale LLM usage selectively rather than system-wide, thus managing complexity while maintaining assistance quality.
Solution Approach 2:
The patent introduces an intermediary layer between the DAP core and LLM engines, where widgets act as mediators that translate user interactions into LLM prompts and process LLM outputs into actionable assistance. This intermediary architecture isolates the complexity of LLM integration within specific components rather than requiring system-wide restructuring.
2Productivity
If LLM engines are used for text analysis and generation in DAPs, then user efficiency in text-related operations is improved, but data privacy risks and computational overhead increase
Solution Approach 1:
The system applies local quality by enabling LLM processing only in specific widgets where text analysis or generation is needed, rather than processing all user data uniformly. This allows selective application of powerful LLM capabilities to enhance user efficiency in text-related operations while limiting exposure of unrelated data to privacy risks.
Solution Approach 2:
The patent extracts sensitive data from the overall data flow and processes it through dedicated widget instances that have controlled access to LLM engines. By separating data extraction and processing into discrete widget operations, the system can apply privacy-preserving measures locally without compromising overall user efficiency.
3Adaptability or versatility
If multiple widgets with different automation processes are defined for the same source data, then the versatility and customization of assistance are improved, but the device complexity and configuration overhead increase
Solution Approach 1:
The patent implements universality by designing widgets as multi-functional components that can handle different automation processes (text analysis, generation, transformation) on the same source data. Each widget is configured with specific parameters to determine its function, allowing a single widget type to serve multiple purposes rather than requiring separate components for each function.
Solution Approach 2:
The system applies dynamics by making widget configurations adjustable and adaptable based on user needs. Widgets can be dynamically configured with different automation processes, source data bindings, and presentation settings, allowing the system to adapt to various assistance scenarios without requiring fixed, hard-coded configurations for each use case.
Data Source
AI summary
A method, system and product comprising: defining an identification event associated with a Graphical User Interface (GUI) of a page of a third-party application; defining source data associated to an automation process; defining that, in response to identifying the identification event, an indication of the automation process is configured to be presented over the GUI; defining that, in response to identifying a trigger event, the automation process is configured to be executed by generating a prompt to a Large Language Model (LLM) engine, the prompt comprises a predefined structure of a text portion and a variable portion that is replaced with the source data every invocation of the trigger event; and defining a configuration for presenting in the GUI a result.