Multi-Agent Policy Training for Autonomous Entity Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicles and cobots face challenges in acting in a coordinated and cooperative manner, as existing technologies fail to effectively train policies that promote synchronized behavior among entities.
Innovation Solution
A method using a centralized training approach with the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to learn state-action functions, allowing entities to exchange information and train policies that foster cooperative behavior, even when applied independently, by considering cumulated rewards and individual incentives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized training is used to train policies for multiple autonomous entities, then cooperative and coordinated behavior is improved, but system complexity and computational requirements increase
Solution Approach 1:
The training system is segmented into multiple independent agents, each with its own policy network and state-action function. Each agent trains independently using centralized information (states and actions of all agents) but executes policies independently, dividing the complex training task into manageable agent-specific modules while maintaining cooperative behavior through shared training data
Solution Approach 2:
A centralized training coordinator acts as an intermediary that collects states and actions from all agents, computes the joint state-action functions, and distributes training signals back to individual agents. This mediator enables centralized training without requiring continuous centralized control during execution, resolving the complexity issue
2Reliability
If state-action functions depend on states and actions of all agents, then coordinated behavior is improved, but computational load and training time increase
Solution Approach 1:
The system pre-computes and stores state-action functions for all possible combinations of agent states and actions during the training phase. This preliminary computation allows the trained policies to be executed efficiently without real-time computation of joint state-action values, reducing training time while maintaining coordination
Solution Approach 2:
The state-action functions are dynamically updated during training based on the current joint state and action of all agents. The system adapts the functions iteratively using reinforcement learning signals, allowing the coordination mechanism to evolve and improve over time while managing computational load through efficient update strategies
3Ease of operation
If autonomous entities act independently after training, then operational simplicity is improved, but cooperative behavior may deteriorate
Solution Approach 1:
Each agent is equipped with a trained policy network that enables it to independently determine its own actions based on its local state and the pre-trained understanding of other agents' behaviors. The agents serve themselves by making independent decisions that are optimized for cooperative outcomes during training, achieving both operational simplicity and reliable cooperative behavior
Data Source
Figure 1a~2
Figure 3
AI summary
A method for determining a set of policies for controlling a corresponding set of different acting entities which is represented by a corresponding set of agents is described. The method comprises: determining a set of actions of the corresponding set of agents using a corresponding set of state-action-functions, respectively; wherein the set of state-action-functions is dependent on a set of states of the corresponding set of agents; determining a set of updated states of the corresponding set of agents based on the determined set of actions; determining a set of instant rewards for the corresponding set of agents based on the set of actions and based on the set of states and/or based on the set of updated states of the corresponding set of agents; and updating the set of state-action-functions based on the set of instant rewards; wherein the set of policies for controlling the corresponding set of acting entities is determined based on the updated set of state-action-functions, respectively.