Multi-Agent Reinforcement Learning Data Search with Episode-Length Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Curiosity-based search techniques in multi-agent reinforcement learning often repeat meaningless learning of similar results due to the large number of states and actions in multi-agent environments, where similar outcomes are common, leading to inefficient data collection.
Innovation Solution
An apparatus and method for searching training data that predicts episode length based on agent states and actions, calculating an intrinsic reward from prediction errors, and using it alongside external rewards for multi-agent reinforcement learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If curiosity-based search technique is used in multi-agent reinforcement learning, then intrinsic reward is provided for state changes, but similar results are repeatedly collected in different states leading to meaningless learning
Solution Approach 1:
The patent changes the parameter for intrinsic reward calculation from state-based prediction error to episode length-based prediction error. By predicting the actual episode length and calculating error based on this temporal parameter, the system identifies meaningful episodes that lead to different outcomes, thereby filtering out redundant training data and improving learning quality while maintaining adequate data quantity.
2Productivity
If curiosity-based search technique is used to increase intrinsic reward for state changes, then frequency of selecting action increases, but repeated collection of similar results occurs
Solution Approach 1:
The patent implements feedback by calculating prediction error between predicted and actual episode lengths, and using this error as intrinsic reward. This feedback mechanism allows the system to learn from the temporal structure of episodes, identifying which actions lead to meaningful variations in episode duration, thereby improving learning efficiency and reducing time spent on redundant experiences.
3Reliability
If multi-agent reinforcement learning is performed with large number of states and actions, then collaboration or competition strategy is optimized, but number of states and actions increases exponentially
Solution Approach 1:
The patent extracts the temporal dimension (episode length) as a key feature for intrinsic reward calculation, separating the complexity of state-action spaces from the reward signal. By focusing on episode length prediction error, the system simplifies the reward structure and enables effective learning despite the exponential growth of states and actions in multi-agent environments.
Data Source
AI summary
Provided is an apparatus for searching for training data of multi-agents, the apparatus including: a prediction module that predicts a current episode length based on states and actions of multi-agents; and a calculation module that calculates an intrinsic reward based on a prediction error of the prediction module, wherein the intrinsic reward is used for multi-agent reinforcement learning together with an external reward according to an environment.