Reinforcement Learning Model Heuristic Biasing for Early Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning models, such as AlphaZero, face challenges in providing early and viable results when deployed in dynamically changing games like collectable card games, as they require extensive training and become outdated with rule changes, making initial deployment ineffective.
Innovation Solution
Combining neural network computations with heuristic values to evaluate actions, allowing for earlier deployment by biasing against detrimental actions and improving initial results, thus reducing training time and increasing usability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a reinforcement learning model is trained extensively before deployment, then the model's accuracy and reliability improve, but the training time and opportunity cost increase, making the model useless for initial deployment in dynamically changing games
Solution Approach 1:
The patent applies preliminary action by pre-computing heuristic values for game states and actions before the reinforcement learning model is trained. These heuristic evaluations are stored and can be quickly retrieved during early deployment, allowing the model to provide viable results immediately without requiring extensive training first. The heuristic values serve as pre-prepared guidance that works effectively during the initial phase before the RL model accumulates sufficient training data.
2Productivity
If a pre-trained reinforcement learning model is used for dynamically changing games, then early deployment is enabled, but the model becomes outdated when game rules change, requiring retraining
Solution Approach 1:
The patent implements dynamics by making the evaluation system adaptable to changing game rules. When game rules change, the heuristic evaluation functions can be updated to reflect new rules, and the reinforcement learning model continues to use these updated heuristics. This dynamic updating capability allows the system to maintain effectiveness without requiring complete retraining, as the heuristic component can adapt to rule changes while the RL model learns from the new environment.
Solution Approach 2:
The patent applies parameter changes by allowing the heuristic evaluation parameters to be modified when game rules change. Instead of retraining the entire RL model, the system updates the heuristic parameters to match new game rules, enabling quick adaptation. The RL model then continues training with these updated parameters, maintaining its learned capabilities while adapting to the new game environment.
3Device complexity
If only neural network computations are used for action evaluation, then the model structure remains simple, but initial results are poor and detrimental actions are not effectively biased against
Solution Approach 1:
The patent merges two evaluation approaches: neural network computations and heuristic evaluations. The system combines the probabilistic predictions from the neural network with the domain knowledge embedded in heuristic functions. This combination allows the system to maintain relatively simple individual components while achieving superior initial action quality, as the heuristic part provides immediate guidance based on game knowledge without requiring complex model architecture.
Data Source
AI summary
Systems and methods for improving initial results of a reinforcement learning model are described herein. In an embodiment, a server computer initiates a reinforcement learning model for a modeled system. While executing the reinforcement learning model, the server computer computes a first result value for a particular action using the reinforcement learning model and a second result value for the particular action using a heuristic separate from the reinforcement model. Based, at least in part, on the first result value for the particular action and the second result value for the particular action, the server computer performs the particular action. The server computer determining a result from performing the particular action and updates the reinforcement learning model.