Generative AI UI Element Detection for Language-Agnostic Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional specialized computer vision algorithms for user interface element detection are language-specific, failing to recognize UI elements in non-English languages.
Innovation Solution
An interface engine integrates generative AI models with robotic process automations (RPAs) to detect and automate user interface elements, processing computer activity patterns across multiple UIs, including screenshots and events, to achieve language-agnostic and context-aware automation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If specialized computer vision algorithms are trained on large numbers of computer screen images to identify UI elements, then detection precision is improved, but adaptability deteriorates because the algorithms become language-specific and cannot recognize UI elements in non-English languages
Solution Approach 1:
The system uses a large language model with prompt translation capabilities to achieve universal UI element detection across multiple languages. Instead of training separate language-specific algorithms, the system translates UI text to a target language, uses a single trained model to detect elements, then translates results back. This makes the detection system adaptable to any language without retraining, resolving the contradiction between precision and adaptability.
Solution Approach 2:
The system introduces prompt translation as an intermediary step between the UI element detection algorithm and the target language UI. By translating prompts to match the language of the UI being analyzed, the system enables a single precision-trained algorithm to work across multiple languages without losing detection accuracy or adaptability.
2Measurement precision
If multiple specialized computer vision algorithms are trained for different languages to maintain detection precision, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The system employs a single universal detection algorithm that works across all languages through prompt translation, eliminating the need to maintain multiple language-specific models. This reduces system complexity while preserving detection precision through the translation intermediary layer.
Solution Approach 2:
The system merges multiple language-specific detection capabilities into a single unified algorithm by combining prompt translation with universal element detection. This consolidation reduces the number of separate components needed while maintaining the ability to detect UI elements in any language with high precision.
3Ease of operation
If conventional computer vision algorithms are used for UI element detection, then ease of operation is maintained, but adaptability deteriorates as they cannot handle contextual variations across different languages and UI attributes
Solution Approach 1:
The system introduces prompt translation as an intermediary that automatically adapts the detection algorithm to different languages and UI contexts. This maintains ease of operation because the translation happens automatically without user intervention, while simultaneously improving contextual adaptability by matching prompts to the specific language and attributes of each UI being analyzed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method is provided. The method is executed by an interface engine implemented as a computer program within a computing environment. The interface engine executes detection and automation of computer activity. The method includes recording the computer activity across user interfaces (Uls) and automatically processing the computer activity utilizing a generative Al model to extract patterns. The patterns include portions of the computer activity that are similar or resilient to changes. The method includes determining existing automations according to the patterns.