Policy improvement method, recording medium, and policy improvement apparatus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning techniques face increased processing load and time due to the need for multiple input determinations when updating policy parameters, particularly as the number of components increases, leading to inefficiencies in optimizing cumulative cost and reward.
Innovation Solution
A policy improvement method that adds perturbations to multiple components of a policy parameter simultaneously, estimates the gradient function of the state value function based on input determinations, and updates the parameter using the estimated gradient, reducing the number of input determinations and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple input determinations are performed to update each component of policy parameters, then the policy optimization accuracy is improved, but the processing load and time increase significantly
Solution Approach 1:
The patent combines multiple gradient estimation operations into a single batch processing step. Instead of performing separate input determinations for each parameter component, the method estimates gradients for all components simultaneously using batch data, thereby reducing the total number of input determinations while maintaining optimization accuracy
Solution Approach 2:
The patent performs preliminary batch processing to pre-calculate gradient information for multiple parameter components at once. By preparing and processing batch data in advance, the method avoids repeated individual input determinations during the optimization process, significantly improving processing efficiency
2Adaptability or versatility
If the number of parameter components increases to improve policy representation, then the policy expressiveness is improved, but the number of input determinations increases proportionally
Solution Approach 1:
The patent merges the gradient estimation process for all parameter components into a unified batch operation. Even as the number of parameter components increases, the method processes all gradients simultaneously in a single batch pass, preventing processing time from increasing proportionally with parameter complexity
Solution Approach 2:
The patent changes the processing mode from sequential individual updates to parallel batch processing. By transitioning to batch-based gradient estimation, the method can handle increased parameter dimensions without proportionally increasing processing time, as all parameter updates are computed together rather than one at a time
Data Source
AI summary
A policy improvement method of improving a policy of reinforcement learning by a state value function, is executed by a computer and includes adding a plurality of perturbations to a plurality of components of a first parameter of the policy; estimating a gradient function of the state value function with respect to the first parameter, based on a result of an input determination performed for a control target in the reinforcement learning, the input determination being performed by using the policy that uses a second parameter obtained by adding the plurality of perturbations to the plurality of components; and updating the first parameter based on the estimated gradient function.