AI Task Automation With UI Annotations for Legacy Workflows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI models are susceptible to hallucinations and errors, requiring users familiar with underlying API infrastructure for task automation, and struggle with integrating with traditional/legacy systems lacking well-defined APIs, leading to inaccurate and inefficient task execution.
Innovation Solution
A task automation system that accepts multi-modal input from users to generate programmatic instructions, annotates user interface elements, and integrates with AI models to automate tasks accurately, even with legacy systems, without requiring users to build API connectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If users create API connectors to enable AI models to perform tasks, then task automation capability is improved, but device complexity and ease of operation deteriorate because users must be familiar with underlying API infrastructure
Solution Approach 1:
The patent introduces a low-code development environment as an intermediary layer between the user and the complex API infrastructure. This mediator translates simple visual workflow definitions into executable API connectors, shielding users from underlying complexity while maintaining full automation capability. The low-code environment acts as a buffer that handles the transformation from high-level task descriptions to detailed API interactions.
Solution Approach 2:
The system enables AI models to perform self-service by automatically generating and executing API connectors based on visual workflow definitions created by users. The AI model autonomously interacts with multiple applications and data sources through the low-code environment, eliminating the need for manual API connector creation and reducing user burden while maintaining high automation levels.
2Productivity
If AI models are used to generate task execution instructions, then productivity is improved, but reliability deteriorates due to hallucinations and errors
Solution Approach 1:
The patent implements feedback mechanisms where the low-code development environment continuously monitors and validates AI-generated instructions against the visual workflow definition and actual system state. When discrepancies or potential errors are detected, the system provides corrective feedback to the AI model, allowing it to learn from mistakes and improve accuracy over time while maintaining high productivity.
Solution Approach 2:
The system performs preliminary validation and verification of AI-generated instructions before execution. The low-code environment pre-checks the validity of generated API calls and workflow steps against the visual definition, catching potential errors before they affect reliability. This preliminary action ensures that only verified instructions are executed, maintaining both speed and accuracy.
3Adaptability or versatility
If traditional legacy systems are integrated with AI models, then adaptability is improved, but device complexity worsens due to lack of well-defined APIs
Solution Approach 1:
The patent segments the integration process into distinct visual components within the low-code development environment. Legacy systems are represented as modular workflow elements that can be independently configured and connected. This segmentation allows complex legacy system integrations to be broken down into manageable, visually-defined steps, reducing overall complexity while maintaining adaptability.
Solution Approach 2:
The low-code development environment provides universal integration capabilities that work across multiple application types and legacy systems through a common visual interface. Rather than requiring separate integration approaches for different systems, the platform offers unified tools and patterns that adapt to various legacy system architectures, simplifying the integration process while expanding versatility.
Data Source
AI summary
Systems and methods are described for automating the performance of a task on behalf of a user that the user would otherwise perform manually or semi-manually. In order to automate the task, a user can provide a task automation system with a description of the steps the user would implement in order to perform the task manually. The description of the steps can be provided to the task automation system as multi-modal input. The task automation system may convert the multi-modal input into tokens and input the tokens to an AI model trained to generate a workflow description. The task automation system may later generate instructions using an AI model based on the workflow description and annotations of the application used to perform the task. The task automation system may then implement the instructions generated by the AI model to perform the task automatically on behalf of the user.