Natural Language UI Automation with LLM Semantic Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing UI automation tools face challenges in scalability, flexibility, and ease of use, particularly for non-technical users, due to the complexity of modern software applications and the need for advanced programming skills to handle changes in UI elements.
Innovation Solution
A method and system that utilizes natural language instructions and large language models (LLM) to semantically select UI objects during runtime, allowing non-technical users to perform automation by demonstration, and adapt to changes in UI elements without coding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional UI automation tools are used with programmatic interaction and stable selector anchors, then automation reliability is improved, but scalability and generalization across different applications deteriorate
Solution Approach 1:
The system changes the fundamental parameter of object identification from programmatic selectors to semantic meanings. Instead of relying on stable but application-specific selector anchors, the system uses semantic identification where objects are recognized by their meaning and function rather than their technical identifiers, enabling generalization across different applications while maintaining reliability
Solution Approach 2:
The system achieves universality by creating a semantic layer that works across multiple application types. The same semantic identification mechanism can recognize UI elements in different applications based on their meaning rather than their specific implementation details, allowing one automation system to handle diverse software environments
2Ease of operation
If visual development environments are used in Low-code and No-Code platforms, then ease of operation is improved, but flexibility and adaptability to complex scenarios deteriorate
Solution Approach 1:
The system introduces semantic identification as an intermediary layer between the user's high-level natural language instructions and the low-level programmatic UI interactions. This semantic layer acts as a mediator that translates intuitive user intentions into precise automation actions, maintaining ease of use while enabling flexibility in handling complex UI scenarios
3Measurement precision
If manual testing and repetitive tasks execution are performed, then accuracy is maintained, but time consumption increases significantly
Solution Approach 1:
The system enables automation to perform self-service by using semantic identification to automatically locate and interact with UI elements without human intervention. The semantic layer allows the automation system to independently understand the application interface and execute tasks autonomously, maintaining accuracy while dramatically reducing time consumption compared to manual testing
Data Source
AI summary
A method and system of UI automation includes receiving a demonstration of an automation to be performed on an application. One or more objects and one or more corresponding labels associated with the demonstration are detected on the application. The demonstration is transformed into one or more natural language instructions. An object is semantically selected during a runtime action based on a large language model (LLM). The semantic selection is reflective of an intention captured in the natural language instruction despite any change in a corresponding label of the object.