Screenshot Sequence Extraction for Cross-Platform RPA Workflows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Robotic Process Automation (RPA) technologies struggle with identifying candidate processes due to variability in User Interface (UI) elements across different platforms, leading to high professional service fees and low Return on Investment (ROI).
Innovation Solution
A system and method for sequence extraction using screenshot images to automatically identify candidate tasks by clustering user actions, labeling variations of the same action, and generating an automated workflow based on extracted sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generic UI element information is used for process identification, then the system can work across different platforms, but the information quality degrades due to variability and noise in UI elements
Solution Approach 1:
The patent captures screenshots (visual copies) of the UI instead of relying on generic UI element information. These screenshots serve as faithful representations that preserve the actual appearance and layout of interfaces across different platforms, eliminating the noise and variability inherent in platform-specific UI element data while maintaining cross-platform compatibility
Solution Approach 2:
The patent replaces the mechanical approach of parsing and interpreting UI element data (which is prone to variability) with an image-based approach using screenshots. This substitution allows the system to work directly with visual representations that are consistent across platforms, improving information quality while maintaining versatility
2Measurement precision
If manual process identification is performed to ensure accuracy, then measurement precision improves, but productivity decreases due to high professional service fees and time consumption
Solution Approach 1:
The patent implements automated clustering algorithms that enable the system to identify and organize repetitive tasks without human intervention. The clustering algorithm automatically groups similar UI actions and sequences, providing accurate process identification while eliminating the need for manual analysis and reducing professional service fees
Solution Approach 2:
The patent introduces an intermediate processing layer that automatically analyzes screenshots and extracts process information using clustering algorithms. This intermediary system bridges the gap between raw screenshot data and actionable process insights, providing both accuracy and efficiency by automating the analysis process
3Measurement precision
If detailed UI element information is captured to improve process understanding, then measurement precision improves, but device complexity increases due to handling platform-specific variations
Solution Approach 1:
The patent uses screenshots as universal copies that capture the complete visual state of the UI without requiring platform-specific parsing logic. This approach simplifies the system architecture by eliminating the need for multiple data collection mechanisms for different platforms while maintaining detailed process understanding through visual information
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A system and method for sequence extraction using screenshot images to generate a robotic process automation workflow is disclosed. The system and method include capturing a plurality of screenshots of steps performed by a user on an application using a processor, storing the screenshots in memory, determining action clusters from the captured screenshots by randomly clustering actions into an arbitrary predefined number of clusters, wherein screenshots of different variations of a same action is labeled in the clusters, extracting a sequence from the clusters, and discarding consequent events on the screen from the clusters, and generating an automated workflow based on the extracted sequences.