ML Workflow Step Prediction for GUI Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developers face challenges in creating computerized workflows using graphical user interfaces due to overwhelming options and the difficulty of adhering to best practices, especially for inexperienced users who struggle with learning the development tool and may not utilize best practices effectively.
Innovation Solution
A machine learning-based system predicts and implements additional steps in a computerized workflow by using a trained model to generate text descriptions of next steps based on patterns learned from training data, converting this text into API calls for automation flow builder applications, reducing the need for manual processing and feature engineering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If developers use a graphical user interface to create computerized workflows, then they can develop automated processes without significant knowledge of computer language, but they face overwhelming options and difficulty in adhering to best practices
Solution Approach 1:
The system enables self-service by allowing the computerized workflow to automatically generate and complete workflow steps without requiring developer intervention for each individual step. The workflow autonomously interacts with the graphical user interface, selecting options and configuring parameters based on learned patterns from training data, thereby reducing the burden on developers while maintaining ease of creation.
Solution Approach 2:
The patent introduces an intermediary component that acts as a bridge between the developer's high-level intentions and the detailed interface operations. This intermediary uses machine learning models to translate developer requirements into specific GUI interactions, filtering out the overwhelming options and presenting only relevant choices, thus reducing interface complexity while preserving ease of workflow creation.
2Reliability
If developers manually specify each workflow step, then they have full control over the workflow design, but the process becomes time-consuming and requires significant expertise
Solution Approach 1:
The system applies preliminary action by pre-training machine learning models on extensive workflow data before deployment. During actual workflow creation, the pre-trained model quickly generates appropriate workflow steps based on the specific context, eliminating the need for time-consuming manual specification while ensuring reliable design control through learned best practices and patterns.
Solution Approach 2:
The patent implements feedback mechanisms where the system continuously learns from workflow outcomes and user corrections. When developers review or modify auto-generated steps, this feedback is used to refine the machine learning model, improving its accuracy over time. This ensures that automation maintains reliable design control while progressively reducing development time through increasingly precise predictions.
3Productivity
If inexperienced users are allowed to create workflows independently, then development speed increases, but the quality and adherence to best practices deteriorates
Solution Approach 1:
The patent replaces the mechanical system of manual workflow configuration with an intelligent system based on machine learning. The ML model automatically generates workflow steps with high quality and best practice adherence, substituting the need for experienced human operators. This enables inexperienced users to create workflows at high speed while maintaining manufacturing precision through the model's learned understanding of proper workflow patterns and best practices.
4Manufacturing precision
If the system provides extensive guidance and validation for each workflow step, then workflow quality improves, but the complexity of the development process increases
Solution Approach 1:
The patent extracts the complex validation and guidance logic from the interactive development process and embeds it within the machine learning model. Instead of presenting developers with extensive external guidance and validation checks that increase process complexity, the system internally incorporates this knowledge, generating accurate workflow steps directly. This maintains manufacturing precision while reducing the apparent complexity for users, as the intelligence is hidden within the automated generation process rather than exposed as manual steps.
Data Source
AI summary
An indication to predict one or more additional steps to be added to a partially specified computerized workflow based at least in part on the partially specified computerized workflow is received. Text descriptive of at least a portion of the partially specified computerized workflow is generated. Machine learning inputs based at least in part on the descriptive text are provided to a machine learning model to determine an output text descriptive of the one or more additional steps to be added. One or more processors are used to automatically implement the one or more additional steps to be added to the partially specified computerized workflow.