Speculative Task Planning With Asynchronous Step Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) and generative AI models experience computational inefficiencies due to their large size and structural complexity, leading to significant latency and long wait times in task planning, especially when performing multi-step tasks.
Innovation Solution
A speculative planning system that utilizes an efficient approximation agent and a more powerful target agent, where the approximation agent generates action steps sequentially, with the target agent asynchronously verifying each step, and incorporates a human-in-the-loop mechanism to ensure accuracy and efficiency through user interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLM-based agents are used to perform multi-step tasks, then reasoning abilities and task completion capability are improved, but computational latency and wait times increase significantly
Solution Approach 1:
The patent segments the task planning process into multiple independent steps that can be executed in parallel. Instead of generating the complete plan sequentially, the system breaks down the planning into discrete steps that can be processed simultaneously, reducing overall latency while maintaining task completion capability.
Solution Approach 2:
The system performs preliminary actions by generating candidate plan steps in advance using parallel processing. Multiple potential next steps are generated simultaneously before selection, allowing the system to prepare multiple options ahead of time rather than waiting for sequential generation, thereby reducing computational latency.
2Measurement precision
If LLM-based agents generate complete thought-process lines of text, then reasoning accuracy is improved, but structural complexity and processing time increase
Solution Approach 1:
The patent extracts and removes unnecessary intermediate thought-process text generation from the system. Instead of requiring complete sequential reasoning traces, the system directly generates and evaluates candidate plan steps, eliminating the complex structural overhead of detailed thought-process lines while maintaining reasoning accuracy through parallel candidate evaluation.
Solution Approach 2:
The system creates multiple copies of candidate plan steps in parallel rather than generating a single sequential thought process. By generating multiple candidate steps simultaneously and selecting the best one, the system achieves reasoning accuracy without the structural complexity of extended text-based thought processes.
3Reliability
If sequential LLM calls are used for task planning, then task accuracy is maintained, but productivity and throughput decrease
Solution Approach 1:
The patent segments the task planning into independent candidate step generations that can be executed in parallel rather than sequentially. Each candidate step is processed independently, allowing multiple LLM calls to occur simultaneously, thereby increasing throughput while maintaining accuracy through subsequent selection and verification of the best candidate.
Solution Approach 2:
The system maintains continuity of useful action by keeping multiple LLM calls active in parallel simultaneously. Instead of waiting for one sequential call to complete before starting the next, the system continues multiple planning actions at the same time, maximizing productivity while ensuring task accuracy through candidate selection.
Data Source
AI summary
The present disclosure generally relates to employing an interactive speculative planning system to complete a plan and execute the steps of a requested task. Systems described herein implement a fast approximation agent and an accurate target agent to generate action steps in response to receiving a request to complete a task. For each task, the approximation agent generates action steps sequentially. Simultaneously, for every step the approximation agent produces, the described system calls the target agent asynchronously to generate the next step, using the current trajectory from the approximation agent as a provisional prefix. For each action step, if the outputs of the approximation agent and the target agent match, the described system continues the process. However, if there is a mismatch, the described system halts the approximation agent, and replaces its output with the target agent's output to ensure performance is not compromised.