Autonomous Software Agent Planning Across Application Interfaces
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computer-executable agents are limited to performing predefined tasks through APIs or initiating web searches, unable to interact with applications to complete complex user requests autonomously.
Innovation Solution
A computer-executable agent utilizing multiple machine learning models and generative models to generate high-level and low-level plans, enabling it to interact directly with applications and web browsers to complete complex tasks without relying on predefined APIs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional agents use predefined APIs and rules, then they can perform specific predefined tasks, but they cannot handle complex tasks requiring multiple application interactions
Solution Approach 1:
The patent introduces a visual interface comprehension module as an intermediary between the user interface and the agent's decision-making system. This module captures screenshots, identifies UI elements through computer vision, and translates visual information into actionable data, enabling the agent to interact with arbitrary applications without predefined APIs
Solution Approach 2:
The patent replaces traditional rule-based and API-dependent mechanical systems with a vision-based perceptual system. Instead of requiring pre-programmed knowledge of application interfaces, the agent uses machine learning models to visually comprehend and interpret UI elements, enabling adaptive interaction with any application interface
2Ease of operation
If agents initiate web searches for unsupported tasks, then they can provide search results, but users must manually perform the requested tasks
Solution Approach 1:
The patent implements a self-service mechanism where the agent autonomously performs tasks by directly interacting with application interfaces. The system captures user goals, plans multi-step actions, executes them through UI element identification and interaction, and verifies completion without requiring user intervention or manual web browsing
Solution Approach 2:
The patent employs a planning module that generates preliminary action sequences before execution. The system breaks down user goals into structured task plans with identified UI elements and interaction steps, preparing the complete action sequence in advance to ensure autonomous task completion
3Loss of information
If agents provide textual responses in chat interfaces, then they can communicate information, but they cannot directly complete tasks requiring application interaction
Solution Approach 1:
The patent merges the chat interface communication capability with direct application interaction capability into a unified agent system. The same agent that generates textual responses also controls the visual interface comprehension and action execution modules, eliminating the gap between information provision and task execution
Solution Approach 2:
The patent creates a universal agent architecture that can both communicate through text and execute tasks through direct application interaction. The system uses a unified goal understanding and planning mechanism that works for both conversational responses and automated task execution, making the agent multi-functional
Data Source
AI summary
Various features pertaining to a computer-executable agent are described herein, where the computer-executable agent is configured to complete a multi-step task requested by a user. Several machine learning models, optionally distributed between a server computing system and a client computing device, are utilized to complete the task. The machine learning models generate a high-level plan that describes steps that are to be performed to complete the multi-step task, and further generate low-level plans that describe, for each step, a sequence of actions to be performed by the computer-executable agent to complete the step.