Pixel-Based GUI Script Playback for Repetitive User Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems require repetitive and tedious user interactions for accessing and managing files or resources, such as checking out files from a repository, which are time-consuming and inefficient, especially for multiple users.
Innovation Solution
A system that generates and executes scripts based on pixel data to automate repetitive user interactions with graphical user interfaces, allowing computing systems to perform actions on a GUI without accessing the underlying application, using a token recording and playback engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If users manually perform repetitive operations on graphical interfaces, then precise control over each action is maintained, but time consumption and labor effort increase significantly
Solution Approach 1:
The system records pixel data from the graphical interface during manual operations and creates a script that copies these actions. The script then automatically executes the same sequence of actions by detecting matching pixels and triggering corresponding user interface actions, thereby automating repetitive tasks and reducing time consumption while maintaining precise control over each action step
Solution Approach 2:
The patent replaces manual mechanical user interactions with an automated system that uses pixel data detection and script execution. Instead of requiring physical mouse clicks and keyboard inputs for each repetitive operation, the system uses computational processes to automatically identify UI elements through pixel matching and execute actions programmatically, substituting mechanical operations with electronic automation
2Extent of automation
If scripts are generated based on pixel data, then automation of repetitive tasks is achieved, but system complexity increases
Solution Approach 1:
The system introduces an intermediary layer between the graphical interface and the automation mechanism. This intermediary captures pixel data from the display, processes it to identify UI elements, and translates visual information into executable scripts. By using pixel data as an intermediary representation, the system simplifies the interaction between the complex graphical interface and the automation engine, making the overall system more manageable despite the added automation capability
Solution Approach 2:
The system creates a simplified representation (copy) of the graphical interface state through pixel data capture. Instead of directly manipulating the complex UI framework, the system works with a simplified pixel-based model that can be processed by the script generation engine. This copying approach isolates the complexity of the graphical interface from the automation logic, allowing the system to maintain high automation capability while keeping the core mechanism relatively simple
3Measurement precision
If pixel-based action identification is used, then accuracy in locating UI elements is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary capture and storage of pixel data during the recording phase, when the user manually performs the desired actions. By pre-processing and storing the pixel sequences in advance, the system eliminates the need for real-time pixel analysis during script execution. This preliminary action allows the playback phase to simply match pre-computed pixel patterns, significantly reducing processing time while maintaining high accuracy in UI element location
Solution Approach 2:
Instead of analyzing every pixel in the graphical interface, the system focuses on capturing and comparing only the specific pixel regions where user interactions occur. By concentrating pixel-based analysis on relevant areas rather than the entire interface, the system achieves sufficient accuracy for UI element identification while minimizing unnecessary computational processing time and resource consumption
Data Source
AI summary
In some disclosed embodiments, a computing device may determine that a script identifies first pixel data and at least one first action associated with the first pixel data, and determine that first pixels being displayed on a screen of the computing device correspond to the first pixel data identified in the script. Based at least in part on the first pixels corresponding to the first pixel data and the at least one first action being associated with the first pixel data in the script, the computing device may take the at least one first action at first coordinates corresponding to a first location on the screen at which of the first pixels are being displayed