Multi-Agent Reward Decomposition for Exploration Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-agent reinforcement learning methods face challenges in efficiently searching for an optimal policy in environments with complex reward structures that include both positive and negative rewards, as they rely solely on global rewards for exploration, which can lead to inaccurate action evaluation and suboptimal data collection.
Innovation Solution
The proposed method involves decomposing the global reward into positive and negative local rewards, training value functions using neural networks, and selecting actions based on these learned functions to improve exploration efficiency in multi-agent environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If agents use only global reward for exploration in multi-agent reinforcement learning, then the method is simple to implement, but the action evaluation accuracy deteriorates in complex reward structures
Solution Approach 1:
The patent segments the global reward into multiple local rewards, each corresponding to different aspects of agent behavior. By decomposing the monolithic global reward signal into finer-grained local rewards, the system achieves more accurate action evaluation while maintaining implementation feasibility through modular reward design.
2Adaptability or versatility
If agents select actions randomly for exploration, then the exploration coverage is comprehensive, but the learning efficiency deteriorates
Solution Approach 1:
The patent applies local quality by assigning different exploration strategies to different agents based on their individual roles and the local reward structures they receive. Each agent performs exploration tailored to its specific function and local objectives, achieving comprehensive coverage while improving overall learning efficiency through specialized rather than uniform exploration behavior.
3Ease of manufacture
If the value function is trained only on global reward, then the training process is straightforward, but the policy search accuracy deteriorates in complex environments
Solution Approach 1:
The patent introduces an additional dimension to the training process by incorporating multiple local reward signals alongside the global reward. This multi-dimensional reward structure provides richer training information that improves policy search accuracy in complex environments while maintaining training simplicity through parallel processing of multiple reward streams.
Data Source
AI summary
Provided is an exploration method based on reward decomposition in multi-agent reinforcement learning. The exploration method includes: generating a positive reward estimation model through neural network training based on training data including states of all agents, actions of all the agents, and a global reward true value; generating, for each of the agents, a first individual utility function based on the global reward true value and generating a second individual utility function using the positive reward estimation model; and determining an action of each of the agents using the first individual utility function and the second individual utility function based on the state of each of the agents.