Adaptive Action Repetition in Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing reinforcement learning methods face challenges in managing frequent action changes, which incur high transaction costs in applications like portfolio management, and are limited to risk-neutral settings, failing to adaptively adjust action repetition lengths effectively in risk-sensitive scenarios.
Innovation Solution
A method for adaptively-repeated action selection in reinforcement learning that computes a utility gap between the best and reference actions, using a return distribution predictor, and adjusts action repetition based on a dynamically set threshold, allowing action changes only when the utility gap exceeds the threshold, thereby reducing unnecessary action changes and optimizing utility in both risk-neutral and risk-sensitive settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If frequent action changes are made to maximize utility in reinforcement learning, then the expected return is improved, but transaction costs increase significantly
Solution Approach 1:
The patent applies dynamics by making the action repetition length adaptive rather than fixed. The system dynamically adjusts the repetition length N based on the computed utility gap between the best action and reference action. When the utility gap exceeds a threshold, the system reduces repetition to allow action changes; when the gap is small, it increases repetition to maintain stability. This dynamic adjustment resolves the contradiction by optimizing both return maximization and transaction cost reduction according to real-time conditions.
Solution Approach 2:
The patent changes the parameter of action repetition length from a fixed value to a dynamically adjusted value based on utility gap computation. By computing the difference between the utility of the best action and reference action, and comparing it against a threshold, the system adapts the repetition parameter to balance exploration (maximizing return) and exploitation (reducing transaction costs). This parameter change enables the system to resolve the contradiction between frequent action changes for higher return and action stability for lower transaction costs.
2Loss of energy
If action repetition length is increased to reduce transaction costs, then transaction cost is reduced, but adaptability to changing conditions deteriorates
Solution Approach 1:
The patent implements feedback by continuously computing the utility gap between the best action and reference action at each time step. This feedback mechanism allows the system to monitor changes in the environment and adjust the action repetition length accordingly. When the utility gap exceeds the threshold, indicating significant change or improvement opportunity, the system reduces repetition to adapt to new conditions. This feedback loop resolves the contradiction by maintaining long repetition lengths for cost efficiency while enabling quick adaptation when conditions warrant change.
3Device complexity
If static repetition length is used to simplify the system, then device complexity is reduced, but performance in risk-sensitive settings deteriorates
Solution Approach 1:
The patent changes the repetition length parameter from static to dynamic based on utility gap computation. This parameter transformation enables the system to adapt to risk-sensitive settings by adjusting repetition length according to the significance of utility differences. The computational overhead of calculating utility gaps and comparing against thresholds is minimal compared to the performance gains in complex, risk-sensitive environments. This resolves the contradiction by showing that the increased complexity is justified and manageable for achieving superior performance.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer system for adaptively-repeated action selection in reinforcement learning. A computer computes utilities for respective candidate actions at a current time step, using a return distribution predictor. A computer computes a utility gap between a utility of a best action at the current time step and a utility of a reference action. A computer computes a threshold at the current time step for the utility gap. A computer determines whether the utility gap is greater than the threshold. In responding to determining the utility gap being greater than the threshold, a computer accepts the best action at the current time step. In response to determining the utility gap being not greater than the threshold, at the current time step, a computer rejects the best action and repeats an action that has been taken at a previous time step.