Decision-Making Model With Action-Segmented Experience Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing reinforcement learning methods suffer from biased experience data accumulation due to limited action selection, difficulty in selecting appropriate uniqueness parameters, and biased storage of experience data, leading to reduced learning opportunities and accuracy, especially for actions with fewer occurrences.
Innovation Solution
A decision-making device that stores experience data in separate storage units based on actions, rewards, or reward calculation expressions, and deletes data based on usage frequency to prevent bias, ensuring balanced learning data selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If experience data is stored in a single storage unit and deleted using FIFO method, then storage management is simple, but bias occurs in experience data where frequently performed actions dominate the storage
Solution Approach 1:
The patent divides the single storage unit into multiple storage units, with each storage unit dedicated to storing experience data for a specific action. This segmentation prevents bias by ensuring that experience data for different actions are stored independently, so that actions with lower frequencies are not overwritten by more frequent actions. The segmentation resolves the contradiction by sacrificing some storage management complexity (now needing to manage multiple units) to gain experience data representativeness.
2Device complexity
If experience data is stored in a single storage unit, then storage structure is simple, but learning opportunities for rare actions are reduced due to overwriting
Solution Approach 1:
By segmenting the storage into action-specific storage units, the patent ensures that experience data for rare actions is preserved and not overwritten by frequent actions. This increases learning efficiency for rare actions, as their experience data remains available for learning. The trade-off is increased storage structure complexity, but this is necessary to maintain learning opportunities for all actions.
3Quantity of substance
If uniqueness parameter is used to delete similar experience data, then storage efficiency improves, but it becomes difficult to select appropriate uniqueness parameter and may still result in bias
Solution Approach 1:
The patent extracts the action identifier from the experience data and uses it as the basis for dividing storage units, rather than using a uniqueness parameter that requires complex calculation and selection. This extraction approach simplifies the operation by providing a clear, objective criterion for storage unit assignment (the action itself), eliminating the difficulty of selecting appropriate uniqueness parameters while maintaining experience data volume through action-specific preservation.
Data Source
AI summary
A decision-making device (2) comprising: an action selection unit (200) for selecting one of a plurality of actions that can be taken in a first state so that an environment performs the selected action; a state acquisition unit (201) for acquiring a second state indicating the state that follows the execution of the action; a reward acquisition unit (202) for acquiring a reward serving as an indicator for indicating whether or not the second state is desirable; and a storage processing unit (203) whereby experience data in which the first state, the action, the second state, and the reward are associated is stored in a storage unit (21) associated with the action, the storage unit (21) being one of a plurality of storage units.