Multi-Agent Policy Training for Cooperative Autonomous Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Autonomous vehicles struggle to act in a coordinated and cooperative manner, as existing methods fail to effectively train policies that foster collaborative behavior among multiple entities.

Innovation Solution

A method using a centralized training approach with the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to learn state-action functions, which consider the actions and states of all entities, incorporating empowerment terms to maximize mutual information and correlation, enabling policies that promote cooperative behavior.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If autonomous vehicles use independent decentralized control policies, then each vehicle can operate autonomously, but coordinated and cooperative behavior among multiple vehicles cannot be achieved

Engineering Contradiction:

Improveautonomous operationVSAvoidcoordinated behavior

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The system segments the control policy into two distinct components: a centralized training phase that learns coordinated behavior patterns, and a decentralized execution phase that operates independently. This segmentation allows each vehicle to have autonomous operation capability while the centralized training ensures coordinated behavior is embedded in the policies.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary centralized training of control policies before actual autonomous operation. During this preliminary phase, the centralized controller learns optimal coordinated behaviors and action correlations among multiple vehicles. These pre-learned coordination patterns are then embedded in the decentralized policies, enabling reliable coordinated behavior during independent operation without requiring real-time centralized control.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If centralized training is used to learn coordinated behavior, then cooperative policies can be developed, but the system complexity and computational requirements increase

Engineering Contradiction:

Improvecooperative behaviorVSAvoidtraining system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system introduces an intermediary centralized training controller that mediates between the need for coordinated behavior learning and the requirement for simple decentralized execution. This intermediary component handles the complex computational tasks of learning action correlations and coordinated patterns, while the final decentralized policies remain computationally simple for real-time autonomous operation.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system creates a simplified copy or representation of the complex coordinated behavior patterns learned during centralized training. Instead of implementing the full complexity of centralized control in real-time, the system copies the essential coordination knowledge into decentralized policies that can be executed with much lower computational requirements during autonomous operation.

Inventive Principle:

Principle #26Copying

3Productivity

If existing training methods are used, then individual vehicle policies can be learned, but policies that foster collaborative behavior among multiple entities cannot be effectively trained

Engineering Contradiction:

Improvetraining efficiencyVSAvoidcollaborative behavior

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements a feedback mechanism during centralized training where the performance and coordination quality of multiple vehicle policies are continuously evaluated. This feedback information is used to iteratively improve the policies, ensuring that collaborative behavior patterns are effectively learned. The feedback loop allows the system to adjust training parameters and policy structures to maximize cooperative behavior while maintaining training efficiency.

Inventive Principle:

Principle #23Feedback

Data Source

PatentEP3961598B1Method and system for enabling cooperative coordination between autonomously acting entities

Publication Date: 2024.10.02 BAYERISCHE MOTOREN WERKE AG

Figure 1a~2
Figure 3

AI summary

A method (300) for determining a set of policies (210) for controlling a corresponding set of different acting entities (100, 110) which is represented by a corresponding set of agents (201, 202) is described. The method (300) comprises: determining (301) a set of actions (212) of the corresponding set of agents (201, 202) using a corresponding set of state-action-functions (230), respectively; wherein the set of state-action-functions (230) is dependent on a set of states (211) of the corresponding set of agents (201, 202); determining (302) a set of updated states (211) of the corresponding set of agents (201, 202) based on the determined set of actions (212); determining (303) a set of instant rewards for the corresponding set of agents (201, 202) based on the set of actions (212) and based on the set of states (211) and/or based on the set of updated states (211) of the corresponding set of agents (201, 202); determining (304) at least one empowerment term which is indicative of mutual information and/or correlation between at least one action (212) from the set of actions (212) and at least one updated state (211) from the set of updated states (211); and updating (305) the set of state-action-functions (230) based on the set of instant rewards and based on the at least one empowerment term; wherein the set of policies (210) for controlling the corresponding set of acting entities (100, 110) is determined based on the updated set of state-action-functions (230), respectively.