Machine Learning Device Corrects Reward for Speed-Adaptive Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning methods struggle to minimize the average error of a control target point with respect to a goal trajectory, especially when speed control is involved, due to the influence of speed changes on the discounted cumulative error.
Innovation Solution
A machine learning device that uses a corrected reward and discount rate, adjusted based on the speed and travel distance of the control target point, to learn a control policy that minimizes the average error by integrating these factors into the reinforcement learning process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If reinforcement learning is performed using rewards calculated based on errors, then control accuracy is improved, but speed changes cause the discounted cumulative error to vary with both error and speed, making it difficult to minimize average error
Solution Approach 1:
The patent changes the parameter used for reward calculation from time-based discounting to distance-based discounting. By using the distance traveled by the control target point as the discounting basis instead of time, the reward function becomes adaptable to speed changes. This allows the reinforcement learning system to maintain consistent evaluation criteria regardless of speed variations, resolving the contradiction between control accuracy and speed adaptability
Solution Approach 2:
The patent introduces distance as an intermediary variable that mediates between time and speed. Instead of directly using time for discounting (which is affected by speed variations), the system uses distance as an intermediate measure that is independent of speed. This intermediary approach allows the reward calculation to remain consistent across different speed conditions while still reflecting control accuracy
Data Source
AI summary
According to an embodiment, a machine learning device is con configured to: acquire observation information including information on a speed of a control target point at a control target time; output control information including information on speed control of the control target point, the control information being determined in accordance with the observation information and a control policy; determine a corrected reward obtained by correcting a reward in accordance with a speed of the control target point included in the observation information, the reward being higher as an error between a value of an evaluation parameter and a goal is smaller, the evaluation parameter being a parameter other than a speed derived from the observation information; and perform reinforcement learning of the control policy based on the observation information and the corrected reward.