Reinforcement Learning Trust Region Threshold Adjustment

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing reinforcement learning technologies face inefficiencies in adjusting the upper limit value (δ) of KL divergence, leading to varying training progress and difficulty in finding an optimal value, which prolongs training time and affects training efficiency.

Innovation Solution

A method for dynamically adjusting the upper limit value (δ) of the KL divergence during policy optimization by observing the difference between policies before and after update, using an information processing apparatus to adjust the threshold within a certain trust region, and optimizing the training process through line search and predetermined value adjustments.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If manual trial-and-error is used to find optimal δ values, then training can be performed with various upper limit values, but training time increases significantly and efficiency decreases

Engineering Contradiction:

Improveability to find optimal δVSAvoidtraining time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system automatically adjusts the δ value based on the observed difference between policies before and after update. The adjustment unit monitors the KL divergence and autonomously modifies the upper limit without requiring manual intervention, enabling the system to self-optimize the trust region parameter during training

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The adjustment unit receives feedback from the observed difference between policies and uses this information to dynamically adjust δ. This closed-loop feedback mechanism allows the system to adapt the upper limit based on actual policy changes, eliminating the need for manual trial-and-error and significantly reducing training time

Inventive Principle:

Principle #23Feedback

2Productivity

If a fixed upper limit δ is used throughout training, then the training process is simple, but training progress varies and optimal performance cannot be achieved

Engineering Contradiction:

Improvetraining efficiencyVSAvoidsimplicity of training process

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The system transitions from a static, fixed δ value to a dynamic, adaptive upper limit that changes during training based on observed policy differences. The adjustment unit continuously monitors the KL divergence and modifies δ accordingly, allowing the trust region to expand or contract based on actual training needs and achieving optimal training progress

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20250348791A1Non-transitory computer-readable recording medium, information processing apparatus, and reinforcement learning method

Publication Date: 2025.11.13 FUJITSU LTD

AI summary

A non-transitory computer-readable recording medium has stored therein a program that causes a computer to execute processing including, in a policy optimization problem in reinforcement learning when a trust region is set and policy update is performed, observing a difference between policies before and after update, and adjusting a threshold of the trust region according to an operation of an algorithm performing policy update such that the observed difference remains within a certain range of the trust region.