Cumulative Reward Prediction via Recursive Matrix Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing prediction models for cumulative rewards struggle with non-stationary values, where hyperparameters change over time, leading to poor performance in predicting future system values, especially in systems like temperature fluctuations in machines or water levels in rivers.
Innovation Solution
A computer-implemented method for training a prediction model using recursive updates of matrices and vectors based on eligibility traces and feature vectors at each time step, employing techniques like Least Square Temporal Difference (LSTD) to stabilize predictions across changing conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a prediction model is trained using traditional reinforcement learning methods, then it can predict cumulative rewards, but it performs poorly when values are non-stationary and hyperparameters change over time
Solution Approach 1:
The patent applies dynamics by making the prediction model adaptive to changing conditions through online learning. The model continuously updates its parameters using recursive least squares temporal difference learning, allowing it to adapt to non-stationary environments where hyperparameters change over time, rather than relying on fixed static training
Solution Approach 2:
The patent changes parameters by using recursive least squares temporal difference learning to dynamically adjust model parameters online. Instead of fixing hyperparameters during training, the system continuously updates parameters like the eligibility trace decay rate and learning rate based on incoming data, enabling adaptation to non-stationary conditions
2Measurement precision
If a prediction model works well for a certain period, then it achieves good prediction accuracy, but it works poorly for another period when conditions change
Solution Approach 1:
The patent implements feedback through online learning where the model continuously monitors prediction errors and adjusts its parameters accordingly. The recursive least squares temporal difference learning uses feedback from actual rewards and eligibility traces to update the prediction model, ensuring it remains accurate over extended periods rather than degrading after initial training
Solution Approach 2:
The patent applies preliminary action by pre-computing and storing eligibility traces during the online learning process. These traces capture the temporal dependencies and are used to efficiently update the prediction model, allowing the system to maintain accuracy without requiring extensive retraining when conditions change
3Device complexity
If traditional prediction methods are used for non-stationary values, then the model can be simple to implement, but it requires extensive retraining and computational resources to maintain performance
Solution Approach 1:
The patent replaces traditional batch training mechanics with online recursive learning mechanics. Instead of repeatedly training the entire model on historical data, the system uses recursive least squares temporal difference learning to incrementally update parameters, significantly reducing computational resources while maintaining performance
Data Source
AI summary
A cumulative reward of a target system type is predicted by training a prediction model by performing an iteration for each time step. The iteration includes recursively updating a matrix by using the weighted difference between an eligibility trace of a current time step and an eligibility trace of a previous time step, recursively updating a vector by using a reward of a subsequent time step and the eligibility trace of the current time step, and recursively updating an eligibility trace of a subsequent time step by using a feature vector of the subsequent time step. Each feature vector is an encoded representation of a state of a training system of the target system type at a corresponding time step. The matrix and the vector are output as the prediction model for estimating the cumulative reward of a target time step of a target system of the target system type.