Crowd-Sourced GUI Automation Script Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Non-expert users face difficulties in performing computer tasks due to inadequate technical support and the inefficiency of existing automation tools, which often require expert programming and are slow to cover a wide range of user issues.
Innovation Solution
A system that enables lay users to collaborate in automating computer tasks by recording and aggregating graphical user interface (GUI) actions, filtering out mistakes, and using crowd-sourced translations to create a database of automated solutions adaptable to various user configurations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If expert-based automation scripts are hand-written to automate computer tasks, then task automation capability is improved, but the speed of covering new tasks and the cost of development deteriorate
Solution Approach 1:
The system records and copies the actual GUI actions performed by users to create automation scripts. Instead of hand-writing scripts, the system observes and replicates user interactions with the computer interface, automatically generating executable action sequences that can be stored and replayed for task automation.
Solution Approach 2:
The system enables users to contribute their own task solutions by recording their GUI actions. Each user's successful task completion automatically generates an automation script that benefits the community, allowing the system to self-populate with automation capabilities without requiring expert programmers for each new task.
2Adaptability or versatility
If multiple user GUI traces are aggregated to build a comprehensive solution database, then task coverage is improved, but the complexity of filtering mistakes and redundancies worsens
Solution Approach 1:
The system uses feedback from multiple user executions to identify and filter out mistakes and redundancies. By analyzing patterns across numerous GUI traces, the system determines which actions are consistent and necessary versus which are errors or unnecessary steps, automatically refining the solution database through iterative feedback from user interactions.
Solution Approach 2:
The system merges multiple GUI traces from different users into a consolidated solution. By combining and comparing action sequences from various users attempting the same task, the system creates a unified, optimized script that incorporates the essential steps while eliminating user-specific variations and errors.
3Ease of operation
If automated solutions are provided that require minimal user input, then ease of operation is improved, but the ability to handle user-specific configurations deteriorates
Solution Approach 1:
The automated scripts perform preliminary actions automatically, executing standard task steps without requiring user input. The system pre-configures and executes routine operations based on recorded GUI traces, handling common scenarios automatically while leaving only exceptional or user-specific steps requiring manual intervention.
Solution Approach 2:
The automation system is designed to be dynamic, adapting its behavior based on the specific user context and configuration. The scripts can adjust their execution path and parameters according to detected system states, allowing them to handle user-specific configurations while maintaining ease of operation through automatic adaptation rather than manual customization.
Data Source
AI summary
A system and method that enables a plurality of lay users to collaborate on automating computer tasks is disclosed. In one embodiment, the system automatically performs these tasks, rather than just documenting how to perform them. The system allows a database of solutions to be built for every important computer task. A key characteristic of this system is that users contribute to this database by simply performing the task. The system records the graphical user interface (GUI) actions as the user performs the task. It aggregates GUI traces from multiple users into a canonical sequence of GUI actions parameterized by user-environment that will successfully accomplish the task on a variety of different configurations. A classifier is used to predict which steps are likely to be misinterpreted and requests human intervention to properly perform them. This process can be done iteratively until the translation is believed to be correct.