UI Control Detection via Sub-control Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional robotic process automation systems face challenges in detecting user interface controls in legacy applications without programmatic access, especially in scenarios where only screen images are available, as existing techniques struggle with variations in resolution and window size, and are limited to detecting a few UI controls.
Innovation Solution
A system and method that detects user interface controls by processing image files to identify sub-control objects, correlating them to form UI control objects, and removing extraneous objects based on confidence scores and overlap, using a trained model that breaks down UI objects into sub-objects to handle invariance parameters effectively, allowing for accurate detection across various applications and environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If template-based methods are used to detect UI controls by comparing pixel values, then detection can be performed on applications without programmatic access, but the method fails on even small variations in application resolution and window size
Solution Approach 1:
The patent segments UI controls into multiple sub-controls (e.g., dividing a button into smaller sub-regions). This segmentation allows the detection system to identify controls even when overall dimensions vary, as the relative positions and characteristics of sub-controls remain consistent. The system correlates these sub-controls to reconstruct the complete control object, making detection robust to resolution and window size variations.
Solution Approach 2:
The patent transforms the detection approach by changing from direct pixel-value template matching to analyzing relative spatial relationships and geometric parameters of sub-controls. By focusing on invariant parameters (relative positions, proportions) rather than absolute pixel values, the system maintains detection accuracy across different resolutions and window sizes.
2Extent of automation
If existing AI and machine learning techniques are used for UI control detection, then some automation capability is achieved, but the techniques struggle with variations in resolution and window size and are limited to detecting only one or two UI controls
Solution Approach 1:
By dividing UI controls into multiple detectable sub-controls, the system can identify and correlate multiple control elements within a single application interface. This segmentation strategy enables the detection of numerous controls beyond the limitation of one or two, as each segmented control can be independently detected and then correlated with others.
Solution Approach 2:
The patent develops a universal detection framework that can handle multiple types of UI controls (buttons, text fields, checkboxes, etc.) through a common sub-control correlation mechanism. This multi-functional approach allows the same detection system to work across diverse control types and application interfaces, significantly expanding the number of detectable controls.
3Ease of operation
If conventional RPA systems are used, then robots can interpret user interfaces and execute commands, but they cannot effectively detect controls in legacy applications without programmatic access or in remote desktop environments
Solution Approach 1:
The patent creates a virtual representation (copy) of the UI control structure by detecting and correlating sub-controls from screen images. This copied model of the control hierarchy allows RPA robots to interact with legacy applications and remote desktop environments as if they had programmatic access, enabling command execution without direct application APIs.
Solution Approach 2:
The sub-control correlation system acts as an intermediary layer between the RPA robot and the legacy application interface. By first detecting sub-controls and then correlating them to identify complete controls, the system mediates the interaction process, enabling robots to operate on applications that would otherwise be inaccessible.
Data Source
AI summary
Computerized detection of one or more user interface objects is performed by processing an image file containing one or more user interface objects of a user interface generated by an application program. Sub-control objects can be detected in the image file, where each sub-control object can form a portion of a user interface object that receives user input. Extraneous sub-control objects can be detected. Sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects can be removed. Sub-control objects in the identified set of sub-control objects can be correlated to combine one or more of the sub-control objects in the identified set of sub-control objects to generate control objects that correspond to certain of the user interface objects of the user interface generated by the application program.