Feedback Coefficient Updates Using TD Error Gradient Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional reinforcement learning techniques face difficulties in efficiently updating the feedback coefficient matrix to optimize the accumulated cost or reward, particularly when determining the appropriate perturbation size or direction, leading to suboptimal policy improvements.
Innovation Solution
A policy improvement method that calculates a partial derivative representing the reaction to perturbations for each component of the feedback coefficient matrix, estimates the gradient function matrix of the state-value function, and updates the matrix to optimize the accumulated cost or reward, using a linear difference equation and quadratic form of immediate costs or rewards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional reinforcement learning techniques are used to update the feedback coefficient matrix, then the policy can be improved based on immediate costs or rewards, but the updating efficiency is low and the time to achieve advantageous states is extended
Solution Approach 1:
The patent segments the gradient calculation process by computing the gradient function matrix once and reusing it for multiple feedback coefficient matrix updates. This segmentation separates the computationally intensive gradient estimation from the iterative policy updates, thereby improving productivity while reducing cumulative time loss.
Solution Approach 2:
The patent performs preliminary calculation of the gradient function matrix before the main policy improvement loop. By pre-computing the gradient information based on the state-value function and perturbation data, the system prepares optimization directions in advance, enabling faster subsequent updates of the feedback coefficient matrix without repeated expensive calculations.
2Measurement precision
If perturbation is added to determine the feedback coefficient matrix, then the policy can be optimized, but it is difficult to determine the appropriate perturbation size or direction
Solution Approach 1:
The patent implements feedback mechanisms where the calculated gradient function matrix informs subsequent perturbation selections. The TD error and gradient information feed back into the policy improvement process, enabling the system to automatically adjust perturbation size and direction based on observed state variations and cost/reward changes, thereby improving measurement precision without significantly increasing complexity.
Solution Approach 2:
The patent dynamically adjusts perturbation parameters (size and direction) based on the calculated gradient function matrix and TD error. By changing perturbation parameters adaptively rather than using fixed values, the system achieves more precise policy optimization while maintaining manageable complexity through systematic parameter adjustment rules.
Data Source
AI summary
A non-transitory, computer-readable recording medium stores a program of reinforcement learning by a state-value function. The program causes a computer to execute a process including calculating a temporal difference (TD) error based on an estimated state-value function, the TD error being calculated by giving a perturbation to each component of a feedback coefficient matrix that provides a policy; calculating based on the TD error and the perturbation, an estimated gradient function matrix acquired by estimating a gradient function matrix of the state-value function with respect to the feedback coefficient matrix for a state of a controlled object, when state variation of the controlled object in the reinforcement learning is described by a linear difference equation and an immediate cost or an immediate reward of the controlled object is described in a quadratic form of the state and an input; and updating the feedback coefficient matrix using the estimated gradient function matrix.