Hierarchical DQN Navigation Across Interactive Web Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning in environments with large state and action spaces and sparse rewards is challenging, particularly for automated assistants navigating interactive web documents, where tasks like booking flights require complex interactions with numerous options and sparse feedback.
Innovation Solution
Implementing deep Q-network agents with hierarchical reinforcement learning and using curriculum learning or a meta-trainer to generate synthetic training examples, augmented by dense potential-based rewards, to train agents to navigate web documents efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If reinforcement learning is used to train a policy for automated navigation in web documents, then the automated assistant can perform complex tasks like booking flights, but the large state and action spaces make learning extremely difficult and time-consuming
Solution Approach 1:
The patent segments the large state and action spaces into smaller, manageable subspaces using hierarchical state representation (grouping DOM elements by functionality) and action decomposition (breaking down navigation actions into atomic operations). This segmentation allows the reinforcement learning agent to learn more efficiently by focusing on smaller subsets of the overall task space.
Solution Approach 2:
The patent applies preliminary action by using pre-computed features and embeddings for web document elements before the reinforcement learning process begins. These pre-processed representations (including element hierarchies, functional groupings, and semantic embeddings) provide a structured foundation that accelerates subsequent learning and reduces the time required to navigate complex web documents.
2Productivity
If reinforcement learning with sparse rewards is used for web navigation, then the agent can learn from minimal feedback, but the majority of episodes generate no signal at all, making learning inefficient
Solution Approach 1:
The patent implements dense reward functions that provide continuous feedback throughout the navigation process, rather than sparse rewards only at the end. The reward function incorporates intermediate milestones such as successfully identifying target elements, completing form fields, and progressing through navigation steps, ensuring that the agent receives informative signals in every episode and can learn more efficiently.
Solution Approach 2:
The patent uses pre-computed goal states and target element identification before the navigation begins. By having预先 identified goals and structured task descriptions, the system can provide more informative and dense reward signals during the learning process, as the agent knows what constitutes progress toward the goal at each step.
3Adaptability or versatility
If the automated assistant needs to navigate through numerous interactive elements to follow dynamic instructions, then it can handle complex user intents, but the number of possible airport/date combinations grows large, making it infeasible to visit each option
Solution Approach 1:
The patent segments the web document into functional regions and groups DOM elements by their purpose (e.g., form fields, navigation elements, content areas). This hierarchical organization allows the agent to navigate to relevant sections directly without exploring every element, significantly reducing the effective state space while maintaining the ability to handle complex instructions.
Solution Approach 2:
The patent applies local quality by creating specialized representations for different types of web elements based on their functionality. Each element type (buttons, inputs, selects) has tailored features and action spaces, allowing the agent to apply domain-specific knowledge and reduce the complexity of navigating through numerous options by understanding the local context and requirements of each element.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure is generally directed to methods, apparatus, and computer-readable media (transitory and non-transitory) for learning to automatically navigate interactive web documents and/or websites. More particularly, various approaches are presented for training various deep Q network (DQN) agents to perform various tasks associated with reinforcement learning, including hierarchical reinforcement learning, in challenging web navigation environments with sparse rewards and large state and action spaces. These agents include a web navigation agent that can use learned value function(s) to automatically navigate through interactive web documents, as well as a training agent, referred to herein as a "meta-trainer," that can be trained to generate synthetic training examples. Some approaches described herein may be implemented when expert demonstrations are available. Other approaches described herein may be implemented when expert demonstrations are not available. In either case, dense, potential-based rewards may be used to augment the training.