UAV Air Combat Maneuvering via Parallel Self-Play Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing air combat maneuvering methods for UAVs, whether traditional or intelligent, face challenges in robustness and adaptability when transitioning from a single deterministic environment to new environments, leading to suboptimal decision-making and prolonged training times.
Innovation Solution
An air combat maneuvering method based on parallel self-play, utilizing a soft actor-critic (SAC) algorithm, constructs UAV maneuver models, state and action spaces, and reward functions to enable self-play training, enhancing confrontation levels and combat success rates through shared strategies across multiple environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If training is performed in a single deterministic environment, then the policy model is highly adaptive to the current environment, but the robustness is weak and the model cannot select rational decision actions accurately in new environments
Solution Approach 1:
The patent transitions from single-environment training to multi-environment parallel training, adding the dimension of environmental diversity. By constructing multiple different battlefield environments and training policy models simultaneously across these environments, the system achieves both adaptability to specific environments and robustness across diverse scenarios.
Solution Approach 2:
The patent creates policy models that serve multiple functions by training them across diverse environments. The same training framework and algorithm structure are universally applied across different battlefield scenarios, enabling the models to develop generalized decision-making capabilities that work across various environments while maintaining specific adaptability.
2Adaptability or versatility
If learning from scratch in a new environment, then the model can adapt to the new environment, but it takes a lot of time for training
Solution Approach 1:
The patent performs preliminary training in multiple diverse environments before deployment. By pre-training policy models across various battlefield scenarios, the system prepares the models with generalized knowledge in advance, so when deployed to a new environment, they can adapt quickly without requiring extensive retraining from scratch.
Solution Approach 2:
The patent uses parallel self-play training where multiple policy models are trained simultaneously across different environments, copying the same training algorithm and framework across multiple instances. This parallel approach accelerates the learning process by distributing training across multiple environments concurrently rather than sequentially.
3Manufacturing precision
If traditional methods using expert knowledge and formula derivation are used, then the decision-making process is mathematically optimized, but it lacks self-optimization process
Solution Approach 1:
The patent implements self-service through reinforcement learning where policy models automatically optimize their own decision-making strategies through interaction with the environment. The models receive rewards or penalties based on their actions and autonomously adjust their policies to improve performance, eliminating the need for manual mathematical optimization while achieving continuous self-improvement.
Solution Approach 2:
The patent incorporates feedback mechanisms through reward functions that provide continuous information to the policy models about the quality of their decisions. This feedback loop enables the models to learn from their actions and outcomes, automatically optimizing their strategies based on environmental responses rather than relying on pre-defined mathematical formulations.
4Extent of automation
If intelligent methods using genetic algorithms and artificial intelligence are used, then the model can self-learning and self-optimization, but the robustness is weak when applied to new environments
Solution Approach 1:
The patent adds the dimension of environmental diversity by training intelligent policy models across multiple different battlefield environments simultaneously. This parallel multi-environment training approach maintains the self-learning capability of intelligent methods while significantly improving robustness by exposing the models to varied scenarios during training, preventing overfitting to any single environment.
Data Source
AI summary
The present disclosure provides an air combat maneuvering method based on parallel self-play, including the steps of constructing a UAV (unmanned aerial vehicle) maneuver model, constructing a red-and-blue motion situation acquiring model to describe a relative combat situation of red and blue sides, constructing state spaces and action spaces of both red and blue sides and a reward function according to a Markov process, followed by constructing a maneuvering decision-making model structure based on a soft actor-critic (SAC) algorithm, training the SAC algorithm by performing air combat confrontations to realize parallel self-play, and finally testing a trained network, displaying combat trajectories and calculating a combat success rate. The level of confrontations can be effectively enhanced and the combat success rate of the decision-making model can be increased.