Reinforcement Learning Value Iteration With Bounded Complex Returns
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Value iteration methods in reinforcement learning struggle to effectively exploit n-step returns due to off-policy bias, as importance sampling methods require knowledge of both target and behavior policies, which is not available in this framework.
Innovation Solution
A bounding method using a negatively biased, low-variance complex return estimator provides a lower bound on traditional one-step returns, and the Bounded-FQI algorithm efficiently incorporates this into an approximate value iteration framework, exploiting off-policy bias in trajectories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If importance sampling methods are used to exploit n-step returns, then the ability to exploit off-policy bias improves, but the requirement for knowledge of both target and behavior policies increases device complexity
Solution Approach 1:
The patent extracts and eliminates the requirement for behavior policy knowledge from the importance sampling framework. By deriving a modified importance sampling estimator that only requires the target policy, the invention removes the complex dual-policy requirement while preserving the ability to exploit off-policy bias in reinforcement learning trajectories.
Solution Approach 2:
The patent changes the parameter requirements of the importance sampling method by modifying the estimator formulation. Instead of requiring both target and behavior policies as inputs, the new estimator changes the parameter set to only require the target policy, thereby reducing device complexity while maintaining productivity benefits.
2Device complexity
If traditional one-step return estimators are used, then simplicity is maintained, but measurement precision of value estimates deteriorates
Solution Approach 1:
The patent introduces dynamic n-step return estimation into the value iteration framework while maintaining practical simplicity. The estimator dynamically adapts the look-back horizon to exploit off-policy bias in trajectories, improving measurement precision without requiring complex dual-policy formulations. The method remains computationally tractable by using only the target policy.
Solution Approach 2:
The patent introduces a modified importance sampling estimator as an intermediary between the simple one-step return and the complex dual-policy requirement. This intermediary estimator maintains simplicity by requiring only the target policy while achieving improved precision through n-step return exploitation with off-policy bias correction.
Data Source
AI summary
A system and method for controlling a system, comprising estimating an optimal control policy for the system; receiving data representing sequential states and associated trajectories of the system, comprising off-policy states and associated off-policy trajectories; improving the estimate of the optimal control policy by performing at least one approximate value iteration, comprising: estimating a value of operation of the system dependent on the estimated optimal control policy; using a complex return of the received data, biased by the off-policy states, to determine a bound dependent on at least the off-policy trajectories, and using the bound to improve the estimate of the value of operation of the system according to the estimated optimal control policy; and updating the estimate of the optimimal control policy, dependent on the improved estimate of the value of operation of the system. The control system may produce an output signal to control the system directly, or output the optimized control policy. The system preferably is a reinforcement learning system which continually improves.