UI-Grounded Task Guidance Using Lookahead Action Planning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional UI virtual assistant systems fail to adapt flexibly to the current user interface, leading to irrelevant, inaccurate, and inefficient instructions for performing tasks, often hallucinating non-existent UI elements and requiring excessive user interactions.
Innovation Solution
A UI-grounded action prediction system using large language models for lookahead plan generation, chain-of-thought reasoning, and cooperative reasoning to generate instructions that are accurately grounded in the current UI elements, separately determining operations and target interactive elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional virtual assistant systems provide task instructions, then users receive guidance on how to perform tasks, but the instructions are often irrelevant to the current user interface and require excessive user interactions
Solution Approach 1:
The system performs preliminary action by generating a lookahead plan that predicts the sequence of actions needed to complete a task before the user executes them. The virtual assistant anticipates user needs by determining what actions will be required next, allowing it to provide proactive guidance rather than reactive responses, thereby reducing the number of interactions needed.
Solution Approach 2:
The system implements feedback by continuously monitoring the current user interface state and comparing it against the predicted lookahead plan. When the actual UI state diverges from expectations or when users request help, the system adjusts its instructions accordingly, providing context-aware guidance that adapts to the actual interface being used rather than relying on static pre-defined instructions.
2Adaptability or versatility
If conventional virtual assistant systems generate instructions based on pre-defined task flows, then task completion guidance is provided, but the instructions fail to adapt to the actual user interface being used
Solution Approach 1:
The system applies dynamics by making the instruction generation process adaptive rather than static. Instead of relying on fixed pre-defined task flows, the virtual assistant dynamically generates instructions based on the current UI state, the predicted lookahead plan, and chain-of-thought reasoning about what actions are appropriate given the actual interface elements present. This allows the system to adapt to different UI configurations while maintaining reliability through reasoned decision-making.
Solution Approach 2:
The system introduces an intermediary layer of chain-of-thought reasoning and lookahead planning between the user's task goal and the generated instructions. This intermediary process analyzes the current UI state, predicts necessary actions, and reasons about appropriate interactions before formulating instructions, thereby bridging the gap between static pre-defined flows and dynamic UI adaptation while ensuring accurate UI element identification.
3Ease of manufacture
If conventional systems use static documentation or pre-defined task flows, then instruction generation is simple, but the instructions become outdated and irrelevant to the current interface
Solution Approach 1:
The system implements self-service by enabling the virtual assistant to autonomously generate context-relevant instructions without relying on static documentation or extensive pre-programming. The lookahead planning mechanism allows the system to independently determine what actions are needed based on the current UI state and task goals, while chain-of-thought reasoning enables it to adapt its instruction generation to the specific interface elements present, making the system self-sufficient and continuously relevant.
Data Source
AI summary
The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate instructions for performing a next action of a task. For instance, in some cases, the disclosed systems receive, from a client device interacting with a software application, a query for performing a task via a user interface of the application. The disclosed systems generate a lookahead prompt having an execution example corresponding to the task, the execution example including an example task and an example action sequence for the example task. The disclosed systems also generate, from the lookahead prompt using a large language model, an estimated lookahead plan describing one or more actions for performing the task. The disclosed systems also use one or more large language models to generate, from the estimated lookahead plan, instructions to perform a next action for the task via user interaction with an interactive element of the user interface.