Agentic Trajectory Generation for Multimodal Workflow Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models require massive amounts of labeled data for training, which is laborious and time-consuming, and there is a high degree of coupling between tasks and data, limiting their performance on new tasks without sufficient data.
Innovation Solution
Integrate human-in-the-loop (HITL) methods to incorporate human knowledge and experience, using core set construction and active learning to select key samples for training, and develop Adept Workflow Language (AWL) for composing multimodal web interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If massive amounts of labeled data are used for training deep learning models, then model performance is improved, but data annotation becomes laborious and time-consuming
Solution Approach 1:
The system performs preliminary actions by using the AI agent to automatically execute tasks and generate synthetic training data before formal model training. The agent explores the software interface, performs actions, and records trajectories that are then used as training data, eliminating the need for manual data annotation while providing sufficient training examples for the model
Solution Approach 2:
The system implements self-service by enabling the AI agent to autonomously generate its own training data through self-directed exploration and task execution. The agent independently interacts with the software interface, captures its actions and outcomes, and creates training trajectories without human intervention, making the data generation process self-sufficient
2Reliability
If deep learning models are trained on extensive labeled data, then task performance is improved, but the coupling between tasks and data increases, limiting adaptability to new tasks
Solution Approach 1:
The system achieves universality by creating a multi-functional training framework where a single AI agent can be trained on diverse software interfaces and task types. The agent learns generalizable skills through varied training trajectories from different applications, enabling it to adapt to new tasks and interfaces without task-specific retraining, thus decoupling performance from task-specific data
Solution Approach 2:
The system implements dynamics by making the training data generation process adaptive and flexible. The AI agent dynamically explores the software interface and generates training trajectories based on the specific task requirements and interface characteristics, allowing the training data to adapt to different tasks rather than requiring static, task-specific labeled datasets
3Reliability
If human-in-the-loop methods are used to incorporate human knowledge, then model performance on new tasks is improved, but system complexity increases
Solution Approach 1:
The system uses an AI agent as an intermediary between human intent and software execution. The agent translates natural language instructions into actionable steps, interacts with the software interface, and returns results, serving as a mediator that incorporates human knowledge and intent while managing the complexity of interfacing with diverse software systems
Data Source
AI summary
A system for generating training data to train agents to automate tasks otherwise done by users includes an intermediary disposed between an interface and a user. The intermediary is configured to: intercept one or more user-actuated actions directed towards the interface by the user, the user-actuated actions, if received by the interface, execute a task on the interface; preserve a state of the interface prior to the execution of the task; translate the user-actuated actions into one or more actuation commands, the actuation commands configured to trigger one or more machine-actuated actions that replicate the user-actuated actions on the interface to cause automation of the task; and generate a training dataset to train an agent to automate the task, wherein the training dataset requires the agent to process, as input, the state of the interface prior to the execution of the task, and to generate, as output, the actuation commands.