Multi-Agent Action Decisions With Direct Message Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-agent reinforcement learning systems face challenges in training neural network models due to indirectly determining communication content using task rewards, leading to increased model complexity and communication overheads, especially in limited bandwidth scenarios.
Innovation Solution
A decision-making method where agents process state information through serial network models, using a reward mechanism that directly evaluates cooperation messages and actions, simplifying the network structure and facilitating direct training of models based on task completion and communication relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If agents use a single neural network model to determine both communication content and actions simultaneously, then the model can directly output both outputs, but the network scale increases significantly and training becomes difficult
Solution Approach 1:
The patent divides the single neural network model into two separate models: a communication content determination model and an action determination model. This segmentation reduces the scale of each individual model, making training more manageable while maintaining the capability to generate both communication content and actions. The communication model processes state information to generate communication messages, while the action model uses both state information and communication messages to determine actions, thereby resolving the training difficulty caused by the original monolithic model.
2Ease of manufacture
If task reward is used to indirectly guide learning of communication information, then the reward mechanism remains simple, but model training becomes more difficult
Solution Approach 1:
The patent introduces a direct feedback mechanism where the communication content determination model receives feedback not only from the task reward but also from the action determination process. The action model's performance and the actual communication effectiveness provide direct feedback signals that guide the communication model's learning. This dual feedback approach maintains the simplicity of the task reward mechanism while significantly easing training difficulty by providing more direct learning signals for communication content generation.
3Ease of manufacture
If communication overheads are not considered in reinforcement learning objective, then the learning objective remains simple, but the dimension of learned communication content becomes large and inapplicable to actual communication scenarios
Solution Approach 1:
The patent modifies the reinforcement learning objective by incorporating communication overhead constraints as an additional parameter. The learning objective now includes both task completion reward and communication cost penalty, creating a composite objective function. This parameter change guides the communication content determination model to generate compact, efficient communication messages that balance task effectiveness with communication resource consumption, making the learned communication content applicable to actual bandwidth-limited scenarios while maintaining learning objective clarity.
Data Source
AI summary
A decision-making method for an agent action and a related device are provided and are used in the field of communication technologies. The method includes: a first agent processes first state information obtained from an environment through a first model, to obtain a first cooperation message; the first agent sends the first cooperation message to at least one second agent; the first agent receives second cooperation message sent by the at least one second agent; the first agent processes the first cooperation message and the second cooperation message through a second model, to obtain a first cooperation action performed by the first agent, where the second cooperation message is sent by the at least one second agent.