Multi-Agent Reinforcement Learning Exploration via Inter-Agent Influence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-agent reinforcement learning faces challenges in efficiently exploring a vast action space due to the exponential increase in action combinations and the lack of exploration techniques that consider the influence between agents, leading to low performance in complex environments.
Innovation Solution
An exploration method and apparatus that calculate the influence of each agent's action on other agents, using a linear sum of utility function values and variance propagation techniques to probabilistically select actions for training samples, enhancing the exploration efficiency by considering inter-agent interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If value function-based methods with individual utility functions are used for exploration, then the method is simple to implement, but it fails to consider inter-agent influence leading to low performance in complex environments
Solution Approach 1:
The patent merges individual utility functions with inter-agent influence by calculating the variance of joint value functions across other agents' actions. This combination allows the exploration method to maintain the simplicity of individual utility evaluation while incorporating the complex interdependencies between agents through variance propagation, thereby resolving the contradiction between ease of implementation and performance in complex environments.
2Productivity
If random exploration or individual utility-based exploration is used, then the exploration process is computationally efficient, but it cannot efficiently explore the vast action space in multi-agent systems
Solution Approach 1:
The patent changes the exploration parameter from simple utility values to variance values of joint value functions. By propagating variance through the action space and using it to guide exploration, the method efficiently identifies promising regions without requiring exhaustive search of the vast action space, thus maintaining computational efficiency while reducing exploration time in multi-agent systems.
3Loss of information
If all agent observation information is used during training (CTDE paradigm), then the available information for learning is maximized, but the action space complexity increases exponentially
Solution Approach 1:
The patent extracts the essential inter-agent influence component from the full joint value function by calculating variance with respect to other agents' actions while keeping the current agent's action fixed. This extraction approach allows the method to utilize all available observation information for training while focusing computation on the relevant variability caused by inter-agent interactions, thereby avoiding the exponential complexity of the full action space.
Data Source
AI summary
An exploration method used by an exploration apparatus in multi-agent reinforcement learning to collect training samples during the training process is provided. The exploration method includes calculating the influence of a selected action of each agent on the actions of other agents in a current state, calculating a linear sum of the value of a utility function representing the action value of each agent and the influence on the actions of the other agent calculated for the selected action of each agent, and obtaining a sample to be used for training an action policy of each agent by probabilistically selecting the action in which the linear sum is the maximum, and the random action.