Reinforcement Learning Policy Update via Generalized Inverse Matrix
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing reinforcement learning methods face increased processing load and time due to the high number of input decisions required when updating policy parameters, especially as the number of parameter components grows, making it inefficient to optimize cumulative cost and reward.
Innovation Solution
A policy improvement method that uses a generalized inverse matrix to update policy parameters based on inputs to the control target, reducing the need to estimate the gradient function matrix of the state value function and thereby minimizing the number of input decisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional reinforcement learning methods are used to update policy parameters, then the policy can be improved based on state value function, but the number of input decisions and processing load increases significantly
Solution Approach 1:
The patent extracts and eliminates the need to estimate the gradient function matrix of the state value function. By using a different mathematical approach that directly computes policy updates from state-value differences without requiring gradient estimation, the method removes this computationally intensive step while maintaining policy improvement accuracy.
Solution Approach 2:
Instead of following the traditional approach of estimating gradients to determine policy updates, the patent inverts the methodology by directly computing updates through alternative mathematical relationships. This reversal avoids the computationally expensive gradient estimation process while achieving the same policy improvement objective.
2Adaptability or versatility
If the number of policy parameter components increases, then the policy can represent more complex behaviors, but the number of input decisions required for updating parameters increases
Solution Approach 1:
The patent removes the dependency between the number of parameter components and the number of required input decisions. By eliminating gradient function matrix estimation, the update process no longer scales with parameter dimensionality in the traditional way, allowing complex policies to be updated efficiently regardless of the number of parameters.
Solution Approach 2:
The patent changes the mathematical parameters and relationships used in policy updates. Instead of using gradient-based methods that require multiple input decisions proportional to parameter count, it employs alternative mathematical relationships that compute updates more directly, reducing the time required even as parameter complexity increases.
Data Source
AI summary
A policy improvement method of improving a policy of reinforcement learning based on a state value function is performed by a computer. The method causes a computer to execute a process including: calculating an input to a control target based on the policy and a predetermined exploration method of exploring for an input to the control target in the reinforcement learning; and updating a parameter of the policy based on a result of applying the calculated input to the control target, using the input to the control target and a generalized inverse matrix regarding a state of the control target.