UI Element Detection Using Unified Series-Parallel Targeting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graphical element detection techniques in robotic process automation (RPA) are not optimal for all scenarios, as they are typically applied individually and lack efficiency in identifying UI elements across different applications and UI types.
Innovation Solution
A combined series and delayed parallel execution unified target technique is employed, which configures default UI element detection methods at the application or UI type level, using a plurality of graphical element detection techniques such as selectors, computer vision, and optical character recognition, to identify UI elements at runtime by analyzing and comparing UI element attributes with descriptor attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple graphical element detection techniques are applied individually, then each technique can be simple and fast, but the overall detection accuracy and reliability are insufficient for all scenarios
Solution Approach 1:
The patent combines multiple graphical element detection techniques (selectors, computer vision, OCR) into a unified detection system that executes them in series and parallel configurations. This merging allows the system to leverage the strengths of each technique while maintaining overall system reliability across diverse UI scenarios.
Solution Approach 2:
The unified target technique creates a multi-functional detection system that can adaptively select and execute appropriate detection techniques based on the specific UI element and scenario. This universal approach enables a single system to handle various detection needs without requiring separate specialized systems for each technique.
2Measurement precision
If a single graphical element detection technique is used, then the system complexity is low, but the detection accuracy varies and is not optimal for all UI scenarios
Solution Approach 1:
The system dynamically selects and executes detection techniques based on real-time conditions and UI element characteristics. The unified target technique adapts the detection approach during runtime, switching between series and parallel execution modes to optimize accuracy for each specific detection scenario while managing complexity adaptively.
Solution Approach 2:
The patent changes the execution parameters of detection techniques by implementing configurable series and parallel execution modes. This allows the system to adjust detection behavior based on scenario requirements, optimizing accuracy by selecting appropriate execution patterns while controlling complexity through parameter management.
3Reliability
If multiple detection techniques are executed in parallel from the start, then detection coverage is comprehensive, but the execution time and resource consumption increase
Solution Approach 1:
The patent segments the detection process into distinct phases: initial series execution of primary techniques, followed by conditional parallel execution of additional techniques only when needed. This segmentation ensures comprehensive coverage while minimizing execution time by avoiding unnecessary parallel processing for elements detected by initial techniques.
Solution Approach 2:
The system implements periodic detection execution with two phases: an initial detection phase using series execution, and a subsequent parallel execution phase triggered only if the initial phase fails to detect the element. This periodic approach balances comprehensive coverage with efficient time utilization.
4Productivity
If graphical element detection is performed without a unified technique, then each detection method can be independently configured, but the overall system lacks coordination and efficiency
Solution Approach 1:
The unified target technique merges multiple independently configurable detection methods into a coordinated system. It combines selectors, computer vision, and OCR techniques under a single execution framework that manages their interaction, improving overall detection efficiency while providing centralized coordination to handle the complexity of multiple techniques.
Data Source
AI summary
Graphical element detection using a combined series and delayed parallel execution unified target technique that potentially uses a plurality of graphical element detection techniques, performs default user interface (UI) element detection technique configuration at the application and/or UI type level, or both, is disclosed. The unified target merges multiple techniques of identifying and automating UI elements into a single cohesive approach. A unified target descriptor chains together multiple types of UI descriptors in series, uses them in parallel, or uses at least one technique first for a period of time and then runs at least one other technique in parallel or alternatively if the first technique does not find a match within the time period.