Reward Calculation Device for Reinforcement Learning Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In reinforcement learning, calculating internal rewards using all state quantities can lead to a large search space and the exploration of unrelated situations, and pre-selecting feature quantities for internal reward calculation results in lost information.
Innovation Solution
A reward calculation device and method that acquires state quantities, selects a subset based on a selection criterion, calculates internal and external rewards, and updates the selection criterion using a multi-armed bandit model to improve calculation efficiency and select valid internal rewards without exploring all combinations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all state quantities are used to calculate internal reward, then the search space becomes large and exploration of unrelated situations occurs, but calculation efficiency deteriorates
Solution Approach 1:
The patent segments the complete set of state quantities into multiple subsets, where each subset contains a portion of the state quantities. Instead of evaluating all state quantities simultaneously, the system divides them into manageable groups that can be processed separately and efficiently.
Solution Approach 2:
The patent dynamically selects which subset of state quantities to use for calculating internal rewards based on current learning progress and performance. The subset configuration is not fixed but adapts over time, allowing the system to adjust the search space coverage according to actual needs.
2Device complexity
If a designer selects feature quantities for internal reward calculation in advance, then calculation complexity is reduced, but information loss occurs
Solution Approach 1:
The system automatically determines the optimal subset of state quantities through self-learning and adaptation, eliminating the need for manual designer selection. The learning apparatus itself performs the feature selection task, using reinforcement learning to identify which state quantities are most valuable for internal reward calculation.
Solution Approach 2:
The patent implements a feedback mechanism where the system evaluates the effectiveness of different state quantity subsets based on learning performance. This feedback is used to iteratively improve the subset selection, ensuring that important information is retained while maintaining computational efficiency.
Data Source
AI summary
A reward calculation device includes: an acquisition portion that acquires a state quantity of an apparatus which performs a task and a state quantity of a target object to which the task is performed; a storage portion that stores a subset of a plurality of state quantities used for an internal reward; a selection portion that selects, based on a selection criterion, one from the subset of the plurality of state quantities stored by the storage portion; a reward calculation portion that calculates the internal reward and an external reward by using the subset and the state quantity; and an update portion that updates the selection criterion of the subset of the state quantity based on the internal reward and the external reward.