Reinforcement Learning Discount Rate Correction for Trajectory Error
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning methods fail to effectively minimize the average error of a control target point's trajectory with respect to a target trajectory when speed control is involved, as the value of the discounted cumulative reward is influenced by both error and speed, leading to suboptimal control policies that either increase or decrease speed to maximize the reward.
Innovation Solution
A machine learning device that uses a corrected discount rate, calculated based on the travel distance of the control target point, to learn a control policy through reinforcement learning, ensuring that the discounted cumulative reward is solely determined by the error and not affected by speed, thereby minimizing the average error of the trajectory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional reinforcement learning methods are used to learn speed control, then the control policy can be learned, but the average error of the trajectory cannot be minimized because the discounted cumulative reward is influenced by both error and speed
Solution Approach 1:
The patent extracts the speed component from the discounted cumulative reward calculation by introducing a corrected discount rate that compensates for speed variations. This separates the error measurement from speed influence, allowing the reward to reflect only trajectory accuracy rather than being confounded by speed changes.
Solution Approach 2:
The patent changes the discount rate parameter from a constant value to a speed-dependent corrected discount rate. This parameter transformation allows the reward function to maintain consistency in evaluating trajectory error regardless of the speed at which the error occurs, thereby resolving the contradiction between learning speed control and minimizing trajectory error.
2Extent of automation
If the discount rate is used to calculate cumulative reward, then reinforcement learning can be performed, but the reward value is affected by speed variations leading to suboptimal control policies
Solution Approach 1:
The corrected discount rate acts as an intermediary between the speed variable and the reward calculation. It mediates the influence of speed on the cumulative reward by compensating for speed variations, thereby preventing speed from directly affecting the reward value and allowing automatic control learning to focus on trajectory accuracy.
3Productivity
If reinforcement learning maximizes discounted cumulative reward, then control policy is learned, but speed may increase or decrease to maximize reward rather than to minimize error
Solution Approach 1:
The patent extracts the speed influence from the reward calculation by using a corrected discount rate that compensates for speed variations. This ensures that the reinforcement learning process maximizes a reward that reflects only trajectory error, not speed, thereby resolving the contradiction between learning convergence and error minimization.
Data Source
AI summary
A machine learning device includes an acquisition module, a first calculation module, a second calculation module, a learning module, and an output module. The acquisition module is configured to acquire observation information including information on a speed of a control target point at a control target time. The first calculation module is configured to calculate a reward for the observation information. The second calculation module is configured to calculate a corrected discount rate obtained by correcting a discount rate of the reward in accordance with a travel distance of the control target point. The learning module is configured to learn a control policy by reinforcement learning from the observation information, the reward, and the corrected discount rate. The output module is configured to output control information including information on speed control of the control target point that is determined in accordance with the observation information and the control policy.