LLM Browser Training with Reward Feedback for Web Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in efficiently training large language models to interact with browsers to accurately obtain webpage addresses corresponding to user queries during webpage navigation tasks, which affects the efficiency of human-machine interaction.
Innovation Solution
A method involving obtaining sample texts and operation procedures, determining reward values through interactions with browsers, and performing reinforcement learning to optimize the language model, ensuring accurate and efficient webpage address retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If reinforcement learning training is performed to improve webpage address retrieval accuracy, then the model's accuracy improves, but the training complexity and time consumption increase
Solution Approach 1:
A sample browser is introduced as an intermediary component to execute operation procedures and return actual webpage addresses. This mediator enables the reinforcement learning system to obtain real feedback without direct complex interactions, simplifying the training architecture while maintaining high accuracy through realistic simulation.
Solution Approach 2:
The system implements a feedback mechanism where the sample browser executes operation procedures generated by the large language model and returns actual webpage addresses. These results are compared with expected addresses to calculate reward values, creating a closed-loop feedback system that guides the reinforcement learning process and improves accuracy iteratively.
2Productivity
If reinforcement learning training is performed to improve human-machine interaction efficiency, then the interaction efficiency improves, but the training time consumption increases
Solution Approach 1:
The system performs preliminary actions by pre-defining operation procedures and using a sample browser to pre-execute these procedures before actual training. This allows the model to learn from pre-computed examples, reducing the time needed during actual training while still achieving high interaction efficiency.
Solution Approach 2:
A sample browser is created as a copy of the actual browser environment. This copy can be trained independently and repeatedly without affecting the real system, allowing extensive training to occur in parallel or beforehand, thus reducing the time cost of actual deployment while maintaining high interaction efficiency.
Data Source
AI summary
A method for training a large language model includes: determining a second operation procedure of a first sample text through a first large language model; obtaining a second webpage address obtained through an interaction between the first large language model and a sample browser based on the second operation procedure; determining a target reward value obtained through the interaction between the first large language model and the sample browser according to the second webpage address and a first webpage address corresponding to the first sample text; and performing a reinforcement learning training on the first large language model according to the target reward value.