CNN UI Element Detection in Virtualized Desktop Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Robotic Process Automation (RPA) techniques struggle to accurately detect user interface elements in virtualized desktop environments due to issues with image streaming, leading to unreliable automation and maintenance challenges, especially with changes in resolution, OS scaling, and UI customization.
Innovation Solution
The use of convolutional neural networks (CNNs) is enhanced through the creation of an augmented dataset by applying techniques like channel randomization, hue shift, and random cropping to improve the detection of graphical elements, allowing for robustness against various UI factors and enabling accurate identification of elements regardless of size or resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RPA techniques use API calls with selectors to identify UI elements, then automation works reliably in native computing systems, but the technique breaks down in virtualized desktop environments where images are streamed
Solution Approach 1:
The patent replaces the traditional API-based mechanical selection system with a vision-based detection system using convolutional neural networks. Instead of relying on programmatic selectors that fail in virtualized environments, the system uses image processing and deep learning to detect UI elements directly from screen captures, enabling cross-environment compatibility while maintaining automation reliability
Solution Approach 2:
The patent transforms the detection approach by changing from coordinate-based selectors to pixel-based image analysis. By converting UI element identification into an image recognition problem with configurable parameters like anchor box sizes, stride distances, and confidence thresholds, the system adapts to different display resolutions and virtualized environments while preserving detection accuracy
2Adaptability or versatility
If OCR and image matching techniques are used for VDE scenarios, then some element detection is possible, but the techniques are insufficiently reliable and cause maintenance issues with minor UI changes
Solution Approach 1:
The patent performs preliminary actions by pre-processing images through normalization, resizing, and augmentation before detection. It also pre-trains CNN models on diverse UI datasets and performs preliminary anchor box generation and filtering, creating a robust detection pipeline that maintains reliability despite UI variations in virtualized environments
Solution Approach 2:
The patent implements feedback mechanisms through confidence thresholding and non-maximum suppression that continuously refine detection results. The system uses loss functions during training that provide feedback on detection accuracy, and employs post-processing feedback loops to filter and rank detected elements, maintaining high reliability even when UI elements undergo minor changes
3Measurement precision
If Faster R-CNN is used for graphical element detection, then accurate detection is achieved, but smaller graphical elements may not have representative pixels by the end of the process
Solution Approach 1:
The patent applies segmentation by dividing the feature extraction process into multiple stages with different anchor box scales. It segments the detection space into various resolution levels and uses multi-scale anchor boxes to ensure small elements are captured at appropriate scales, preventing loss of representative pixels during processing
Solution Approach 2:
The patent adds dimensional diversity by implementing multi-scale feature pyramids and varying anchor box aspect ratios. This dimensional approach ensures small graphical elements are detected across multiple scale dimensions, maintaining their representative pixels throughout the detection process while preserving overall detection accuracy
4Ease of manufacture
If standard CNN training is used with limited datasets, then training is faster and simpler, but the model lacks robustness to changes in resolution, OS scaling, and UI customization
Solution Approach 1:
The patent performs preliminary data augmentation by generating synthetic training samples with varied resolutions, colors, and transformations before actual training. This pre-processing step creates a more diverse training dataset that improves model robustness to UI variations while maintaining training efficiency through systematic augmentation rather than collecting diverse real samples
Data Source
AI summary
Graphical elements in a user interface (UI) may be detected in robotic process automation (RPA) using convolutional neural networks (CNNs). Such processes may be particularly well-suited for detecting graphical elements that are too small to be detected using conventional techniques. The accuracy of detecting graphical elements (e.g., control objects) may be enhanced by providing neural network-based processing that is robust to changes in various UI factors, such as different resolutions, different operating system (OS) scaling factors, different dots-per-inch (DPI) settings, and changes due to UI customization of applications and websites, for example.