Autonomous Web Agents Using MCTS, Self-Critique, and RL
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) struggle with generalizing to multi-step reasoning tasks in interactive environments like web navigation due to pre-training on imitation learning datasets that lack necessary behaviors for interactive decision-making, leading to sub-optimal policies and compounding errors.
Innovation Solution
A model is developed that combines search, self-critique, and reinforcement learning, using Monte Carlo tree search (MCTS) for inference-time search and AI self-critique, with Direct Preference Optimization (DPO) for offline training, to improve planning and reasoning in web agents, allowing them to learn from both successful and unsuccessful trajectories with limited supervision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If supervised fine-tuning on curated expert demonstrations is used, then the model learns interactive decision-making behaviors, but it suffers from compounding errors and yields sub-optimal policies due to limited exploration data
Solution Approach 1:
The system performs preliminary exploration using Monte Carlo Tree Search (MCTS) to generate diverse trajectories before formal training. This preliminary action creates a richer dataset that prevents compounding errors by providing multiple valid paths, thereby improving policy optimality without sacrificing exploration capability
Solution Approach 2:
The model performs self-critique and self-evaluation during the training process, automatically generating feedback without external supervision. This self-service mechanism allows the system to identify and correct sub-optimal behaviors while maintaining exploration diversity, resolving the contradiction between policy quality and exploration breadth
2Ease of operation
If Monte Carlo tree search with AI self-critique is used for data collection, then the model improves planning and reasoning, but the training complexity and computational resources increase
Solution Approach 1:
The system introduces an intermediary self-critique module that bridges MCTS exploration and policy training. This intermediary component generates structured feedback from exploration trajectories, enabling improved planning and reasoning while managing training complexity through modular architecture rather than monolithic complexity
Solution Approach 2:
The training process is segmented into distinct phases: MCTS-based exploration, self-critique feedback generation, and preference optimization training. This segmentation allows each component to be optimized independently, improving planning capabilities without proportionally increasing overall system complexity
3Measurement precision
If the model is trained on imitation learning datasets, then it achieves good performance in natural language tasks, but it struggles with generalizing to multi-step reasoning tasks in interactive environments
Solution Approach 1:
The system changes the training data parameters from static imitation learning datasets to dynamic MCTS-generated trajectories with self-critique annotations. This parameter change enables the model to maintain natural language accuracy while gaining multi-step reasoning capabilities through structured interaction data with varying complexity levels
Data Source
AI summary
Systems and methods include improved planning and reasoning capabilities of a web agent, which interacts with a real world website. The agent improves with autonomous experience and requires limited supervision. An MCTS-based search routine is deployed over web pages, a base model is used as an initial distribution, and a number of rationales and possible web actions are explored. AI feedback and self-criticism are used, and the model provides self-evaluation at each node.