Multi-unmanned aerial vehicle cooperative control method and device based on neural network, and medium

Abstract

The present application relates to the field of deep learning, and discloses a multi-unmanned aerial vehicle cooperative control method and device based on a neural network, and a medium. The method comprises: for each formation mode, determining a topology corresponding to the formation mode; generating a corresponding topological matrix on the basis of information flow directions between nodes in the topology; generating a graph convolutional neural network; by means of a backbone network, extracting a multi-dimensional flight feature corresponding to an unmanned aerial vehicle formation; by means of a graph convolution module, outputting a corresponding feature vector on the basis of the multi-dimensional flight feature and an adjacent matrix; by means of an attention module, outputting a corresponding attention feature on the basis of the feature vector, and on the basis of the attention feature, obtaining a feature matrix corresponding to the unmanned aerial vehicle formation; and on the basis of the feature matrix, outputting a cooperative control instruction corresponding to the unmanned aerial vehicle formation. The graph convolutional neural network can generate a more precise and efficient cooperative control instruction, thereby expanding a sensing range and improving the cooperative efficiency of the formation.

Description

Methods, equipment, and media for multi-drone cooperative control of unmanned aerial vehicles based on neural networks

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411719610.7, filed on November 28, 2024, entitled "Method, Device and Medium for Multi-Aircraft Cooperative Control of Unmanned Aerial Vehicles Based on Neural Networks", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of deep learning, specifically to a method, device, and medium for multi-drone formation cooperative control. Background Technology

[0004] With technological advancements, multi-drone swarm technology has demonstrated broad application potential across various fields. By coordinating the work of multiple drones, mission efficiency and success rates can be significantly improved, reconnaissance and surveillance capabilities enhanced, endurance extended, and anti-interference capabilities strengthened. For instance, in some sectors, it can improve the efficiency and effectiveness of logistics and distribution, agricultural plant protection, and environmental monitoring. Furthermore, drone swarms can bring spectacular visual effects to the performance field and play a vital role in areas such as resource exploration and forest fire prevention and disaster relief.

[0005] However, in practical applications, multi-drone swarm technology not only requires close coordination among the drones, but also needs to cope with various complex environmental conditions, such as wind changes, obstacle avoidance, and potential communication interference. Traditional methods, such as rule-based control algorithms or model-based optimization methods, often struggle to simultaneously meet the requirements of positioning accuracy, communication efficiency, perception capabilities, and dynamic response.

[0006] Specifically, the limitations of traditional methods may include:

[0007] 1. Limited perception range: The perception range of drones is usually limited by the type and performance of sensors. In complex environments, such as dense building clusters or complex terrain, the perception range of drones may be insufficient to cover all potential obstacles.

[0008] 2. Limited dynamic obstacle avoidance capability: Although UAVs have a certain dynamic obstacle avoidance capability, in high-speed flight or complex environments, the computational complexity and real-time requirements of obstacle avoidance algorithms may exceed the processing capacity of UAVs, leading to obstacle avoidance failure.

[0009] 3. High algorithm complexity: The formation control algorithm needs to handle the cooperative flight of multiple drones, resulting in high algorithm complexity. In scenarios with high real-time requirements, the algorithm's computational speed and optimization level may be insufficient to meet the demands.

[0010] 4. Insufficient robustness: In complex environments, such as strong winds and rain, the robustness of the formation control algorithm may be challenged, leading to unstable or uncontrollable drone formation flight. Summary of the Invention

[0011] To address the aforementioned problems, this application proposes a neural network-based multi-drone cooperative control method, comprising:

[0012] Based on the current drone application scenarios, we determine the various formation methods required for drone formations, and for each formation method, we determine its corresponding topology.

[0013] Based on the information flow direction between nodes in the topology, a corresponding topology matrix is ​​generated;

[0014] Generate a graph convolutional neural network; the graph convolutional neural network includes at least a backbone network, a graph convolutional module, and an attention module;

[0015] Multi-dimensional flight features corresponding to drone formations are extracted using a skeleton network.

[0016] The topology matrix is ​​used as the adjacency matrix, and the graph convolution module is used to output the corresponding feature vector based on the multi-dimensional flight features and the adjacency matrix.

[0017] The attention module outputs the corresponding attention features based on the feature vectors, and obtains the feature matrix corresponding to the drone formation based on the attention features.

[0018] Based on the feature matrix, output the cooperative control commands corresponding to the UAV formation.

[0019] On the other hand, this application also proposes a multi-drone cooperative control device based on neural networks, comprising:

[0020] At least one processor; and,

[0021] A memory that is communicatively connected to at least one processor; wherein,

[0022] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform, for example, the neural network-based multi-drone cooperative control method described above.

[0023] On the other hand, this application also proposes a non-volatile readable storage medium storing computer-executable instructions, which are set as: the neural network-based multi-drone cooperative control method of the above example.

[0024] The neural network-based multi-drone cooperative control method proposed in this application can bring the following benefits:

[0025] By using deep learning and analyzing the topology and dynamic characteristics of UAV formations, graph convolutional neural networks (Graph Convolutional Neural Networks) can generate more precise and efficient cooperative control commands, thereby improving the perception range and the cooperative efficiency of the formation. In complex environmental conditions, Graph Convolutional Neural Networks can capture and respond to various changes in real time, ensuring that the formation maintains a stable flight state. As a data-driven model, Graph Convolutional Neural Networks can be continuously optimized and adapted to different formation sizes and mission requirements through training, exhibiting strong scalability and adaptability. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0027] Figure 1 is a flowchart illustrating the multi-drone cooperative control method based on neural networks in an embodiment of this application.

[0028] Figure 2 is a schematic diagram of the topology in one embodiment of this application;

[0029] Figure 3 is a schematic diagram of the architecture of a graph convolutional neural network in one embodiment of this application;

[0030] Figure 4 is a schematic diagram of a multi-drone collaborative control device based on neural networks in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0033] As shown in Figure 1, this application embodiment provides a multi-drone cooperative control method for unmanned aerial vehicles (UAVs) based on neural networks, including:

[0034] S101: Based on the current drone application scenarios, determine the required formation methods for drone formations, and for each formation method, determine its corresponding topology.

[0035] Generally, different formation methods are required to meet the needs of different drone application scenarios, and each formation method corresponds to a topology. For each topology, a separate graph convolutional neural network is trained to obtain its own corresponding graph convolutional neural network, thus ultimately resulting in multiple graph convolutional neural networks.

[0036] The application of drone swarm flight technology is becoming increasingly widespread in both military and civilian fields. In the military, it can be used for various missions such as reconnaissance, strike operations, and communications relay. In the civilian sector, it can be used for tasks such as logistics delivery, agricultural protection, environmental monitoring, and drone performances.

[0037] For a drone application scenario, the formation may need to be changed during mission execution based on the requirements. At this time, there are multiple formation modes, and each formation mode has its own corresponding topology. When switching formation modes, the topology and the corresponding graph convolutional neural network also need to be switched.

[0038] In a drone swarm, each drone can be considered a node, and these nodes exchange information via wireless communication links. In the topology, a node represents a drone, and nodes are connected by directed edges. The direction of the edge represents the direction of information flow, referring to the path and direction of information transmission from one node to another. In a drone swarm, the direction of information flow is usually consistent with the direction of communication, meaning information is transmitted from one drone node (sender) to another drone node (receiver).

[0039] For drone swarms, communication between nodes helps determine their own positions. By receiving position information from other drones, drones update their own position estimates, thus achieving more accurate positioning. This communication helps drones maintain accurate positioning in complex environments, avoiding collisions and getting lost. It also helps maintain formation; through communication, drones can learn about the positions and speeds of other drones in real time, adjusting their flight status to maintain formation and improve flight efficiency. Furthermore, it enables collaborative mission completion. Through communication, drones can share mission information, resource allocation, and coordination strategies, thus completing tasks more efficiently. For example, in search and rescue missions, drones can collaboratively search target areas, improving search efficiency; in reconnaissance missions, drones can share reconnaissance information, enhancing overall reconnaissance capabilities.

[0040] As shown in Figure 2, an exemplary topology is provided, in which node 1 has an out-degree of 0, only receives information and does not send information, while node 5 has an out-degree of 4, only sends information and does not receive information. Therefore, in this drone formation, the drone corresponding to node 5 is the most important, and the drone corresponding to node 1 has the least impact on the formation.

[0041] S102: Generate the corresponding topology matrix based on the information flow direction between nodes in the topology.

[0042] Each node is pre-numbered to facilitate subsequent node management. Taking Figure 2 as an example, the node numbers here include: Node 1 to Node 5.

[0043] Nodes are selected sequentially according to their corresponding node numbers, and each node is assigned a single row of parameters in the topology matrix. For the topology structure in Figure 2, nodes 1 through 5 are selected sequentially. Node 1 corresponds to the parameters in the first row of the topology matrix, and so on, with node 5 corresponding to the parameters in the last row of the topology matrix.

[0044] In each row of parameters, nodes other than the current node are selected sequentially according to their corresponding node numbers, and corresponding parameters are generated based on the information flow direction between the current node and other nodes. Here, it is assumed that the parameter is 1 when a node receives information from other nodes, and 0 otherwise; therefore, all parameters on the diagonal of this topology matrix are 0. In the first row of parameters, corresponding to node 1, which receives information from the other four nodes (nodes 2 to 5), the remaining four parameters are all 1. At this point, the first row of the topology matrix is ​​[0, 1, 1, 1, 1]. Similarly, the parameters for each remaining row can be obtained, and the final topology matrix corresponding to the topology structure in Figure 2 can be represented by Formula 1:

[0045] Where A is the topological matrix.

[0046] S103: Generate a graph convolutional neural network; the graph convolutional neural network includes at least a backbone network, a graph convolutional module, and an attention module.

[0047] As shown in Figure 3, the Graph Convolutional Network (GCN) in this application mainly consists of three parts: a backbone network, a graph convolution module, and an attention module. Each part performs its corresponding function.

[0048] The parameters of the graph convolutional neural network have been optimized and adjusted for key parameters of drone flight. Parameters of multiple dimensions are selected as input to the graph convolutional neural network, thereby outputting a corresponding feature matrix to characterize the parameters of the drone formation, thus facilitating the control of the drone formation.

[0049] Specifically, during the optimization process, the following dimensions of flight information (also known as multi-dimensional flight information) can be selected as input to the graph convolutional neural network. These dimensions include: the drone's speed, acceleration, altitude, heading angle, pitch angle, roll angle, wind speed, and obstacle position. Typically, the calculation focuses on the multi-dimensional flight information of individual drones within the drone formation, extracting their corresponding multi-dimensional flight features. These features are then combined with the multi-dimensional flight features of other drones to achieve coordinated control of the drone formation.

[0050] Through model training, multi-dimensional flight information was precisely matched and mapped with the parameters of the graph convolutional neural network (GCNN), achieving a direct correspondence between the GCNN parameters and the UAV flight parameters. This ensures that the model parameters can be updated in real time and accurately, improving the efficiency and accuracy of the GCNN in processing UAV flight data. During parameter matching, the dynamic characteristics of UAV flight and its high real-time requirements were fully considered, ensuring that the GCNN can quickly adapt to changes in the flight environment.

[0051] The training parameters can be obtained by collecting historical data from each drone formation and then performing preprocessing and annotation.

[0052] S104: Extract multi-dimensional flight features corresponding to UAV formations through skeleton networks.

[0053] Backbone networks can take various forms, such as ResNet (Residual Network) and VGG (Visual Geometry Group Network), and are mainly used to match the multi-dimensional flight information of UAVs with the input shape of graph convolutional neural networks.

[0054] Specifically, multi-dimensional flight information is input into the skeleton network for feature extraction. The last layer of the skeleton network is a fully connected layer (Dense layer), which is referred to here as the first fully connected layer.

[0055] At this point, the first fully connected layer of the skeleton network matches the multi-dimensional flight information corresponding to the input drone formation with the shape of the input graph convolution module.

[0056] The formula for the first fully connected layer is shown in Formula 2: y = W0 T Formula 2: x+b0;

[0057] x represents the input multidimensional flight information, y represents the output multidimensional flight features, W0 is the weight matrix, and b0 is the bias matrix.

[0058] For the first fully connected layer, its formula is relatively fixed. In order to ensure that the output of the multi-dimensional flight features conforms to the corresponding shape (that is, the shape corresponding to the multi-dimensional flight features), the matrix size of the weight matrix W0 is set to (n-1)×n, the matrix size of the bias matrix b0 is set to 1×n, and the matrix size of the input multi-dimensional flight information x is set to n×n; where n is the number of dimensions of the multi-dimensional flight information.

[0059] Taking the eight dimensions mentioned above as an example, n=8, the weight matrix W0 is 7×8, and the bias matrix b0 is 8×1.

[0060] After multiplying the weight matrix W0 with the multidimensional flight information x, it is concatenated with the flipped bias matrix b0 to obtain the concatenated vector S; where the size of the concatenated vector S is n×n.

[0061] Taking the eight dimensions mentioned above as an example, the matrix obtained by multiplying the weight matrix W0 with the multidimensional flight information x is still 7×8. The size of the flipped bias matrix b0 is 1×8. The size of the concatenated vector S obtained by concatenating the two is 8×8, which matches the input multidimensional flight information and the input of the graph convolution module.

[0062] Divide the concatenated vector S into multiple sub-vectors Si. i Each sub-vector corresponds to one dimension of the multi-dimensional flight feature; where, sub-vector S i The size of the matrix is ​​n×1.

[0063] Continuing with the example of 8 dimensions mentioned above, each subvector S i The matrix size is 8×1, and each sub-vector corresponds to one of the following dimensions of the UAV: ​​speed, acceleration, altitude, heading angle, pitch angle, roll angle, wind speed, and obstacle position information.

[0064] S105: The topological matrix is ​​used as the adjacency matrix, and the graph convolution module outputs the corresponding feature vector based on the multi-dimensional flight features and the adjacency matrix.

[0065] Specifically, each subvector S i The input is fed into the graph convolution module, and the output is the corresponding sub-feature vector T. i , sub-feature vector T i By concatenating the components, we obtain the corresponding feature vector T.

[0066] The formula for the graph convolution module is shown in Formula 3:

[0067] Among them, w tThis is represented as the output of the graph convolution module, corresponding to the sub-feature vector T. i w s This is represented as the input to the graph convolution module, corresponding to the subvector S. i , Let H be the adjacency matrix. (0) and H (1) Here are the parameters for the graph convolution module, and ReLU is the activation function.

[0068] Wherein, parameter H (0) and parameter H (1) The formula is shown in Formula 4:

[0069] Among them, w (l) Let b be the weight of the l-th layer. (l) For the bias of the l-th layer, H is the adjacency matrix, σ is the activation function; (l) This represents the node characteristics of the l-th layer. When l = 0 or 1, it corresponds to H. (0) or H (1) .

[0070] Taking the eight dimensions mentioned above as an example, the sub-vectors S1 to S8 are input into the graph convolution module, and the output is the sub-feature vectors T1 to T8. These are then concatenated to obtain the feature vector T, which has a size of 8×8. The feature vector T is then input into the attention module for further processing.

[0071] S106: Through the attention module, the corresponding attention features are output based on the feature vector, and the feature matrix corresponding to the UAV formation is obtained based on the attention features.

[0072] Specifically, the formula for the attention module is shown in Formula 5: M s =σ(f 5×5 Formula 5; ([AvgPool(F′);MaxPool(F′)]))

[0073] Where σ is the Sigmoid function (S-shaped function), f 5×5 For a 5×5 convolution process, M s Indicates attentional characteristics;

[0074] [AvgPool(F′);MaxPool(F′)] is a channel concatenation of two feature matrices, AvgPool and MaxPool, respectively, using average pooling and max pooling.

[0075] The parameters of both feature matrices are H×W×C, where H is the height of the feature matrix, W is the width of the feature matrix, and C is the number of channels of the feature matrix. The two feature matrices are concatenated by channels to obtain the concatenated feature parameters H×W×2C.

[0076] S107: Based on the feature matrix, output the cooperative control commands corresponding to the UAV formation.

[0077] After passing through the attention module, the features are input into the second fully connected layer, whose output parameter is set to 8. The resulting output is a feature matrix of size 8×1. Taking the multi-dimensional flight information example above, the output can be directly mapped to the key flight parameters of the UAV, including speed, acceleration, altitude, heading angle, pitch angle, roll angle, wind speed, and precise obstacle location information.

[0078] This information forms the basis for collaborative control decisions among drones. Based on this information, the flight control system can generate a series of collaborative control commands designed to optimize and regulate the flight path of each drone, flexibly adjust its flight speed, and precisely control its flight attitude. These commands are designed to ensure that the drone formation maintains a high degree of coordination and consistency under all conditions, whether maintaining a tight formation or rapidly responding to dynamic changes in the external environment. In this way, the drone formation can not only operate efficiently in complex and ever-changing environments but also effectively avoid obstacles, ensuring the continuity and safety of mission execution, demonstrating a highly intelligent collaborative flight control capability.

[0079] Specifically, the cooperative control commands adjust the drone's current speed, acceleration, altitude, heading angle, pitch angle, roll angle, and other parameters based on the parameters output by the graph convolutional neural network to ensure flight stability.

[0080] When conducting collaborative control of drone formations, different control methods can be selected based on actual needs. For example, it can be the leader-follower method or distributed collaborative control. This requires different modifications for different environments. The topology matrix is ​​mainly determined based on the number of drones in the formation and the direction of information flow between drones.

[0081] Among them, the navigator-follower method involves designating one or more drones as navigators, with the others acting as followers. The navigator is responsible for planning the flight path and speed, while the followers adjust their flight status based on the navigator's information. The distributed cooperative control method uses predefined behavioral rules to distribute and control the movement of drones, achieving formation control.

[0082] Of course, this could also include virtual navigation, where no actual lead drone exists; instead, a virtual navigation point guides the formation flight. A consistency-based control method virtualizes multiple drones into a single structure, controlling the drone formation by controlling this structure.

[0083] By using deep learning and analyzing the topology and dynamic characteristics of UAV formations, graph convolutional neural networks (Graph Convolutional Neural Networks) can generate more precise and efficient cooperative control commands, thereby improving the perception range and the cooperative efficiency of the formation. In complex environmental conditions, Graph Convolutional Neural Networks can capture and respond to various changes in real time, ensuring that the formation maintains a stable flight state. As a data-driven model, Graph Convolutional Neural Networks can be continuously optimized and adapted to different formation sizes and mission requirements through training, exhibiting strong scalability and adaptability.

[0084] In one embodiment, when determining the topology matrix, the number of drones corresponding to each formation can be determined, as well as the center node and edge nodes in the formation. A center node is one in which other nodes exist in all directions around it, while an edge node is one in which other nodes exist only in a certain angular range around it.

[0085] If the number of drones is below a first preset threshold, it indicates that the current drone formation is small, and in this case, selecting a few main drones to send information outwards can simplify the communication structure, improve communication efficiency, and reduce costs. Therefore, among the edge nodes, a preset number of nodes (usually 1 / 5 to 1 / 10 of the drone formation number, rounded down) are selected as the lead nodes to ensure that the lead nodes have the highest out-degree value in the topology.

[0086] As mentioned above, nodes with higher out-degrees are relatively more important, while edge nodes are generally less affected by communication interference. Therefore, one or more edge nodes with the highest out-degrees are selected as lead nodes to guide the drone formation.

[0087] If the number of drones exceeds the first preset threshold, it is considered that the current drone formation has a large number of drones. The fact that each drone sends information to the other drones can provide better robustness and real-time performance. Therefore, the difference between the out-degree values ​​of all nodes in the formation is made lower than the second preset threshold. At this time, the out-degree values ​​of each node are not much different, and they can communicate with each other to increase robustness.

[0088] In one embodiment, during actual application scenarios, formation switching may occur, requiring corresponding topology and graph convolutional neural networks to be switched. During this switching process, differences in parameters within different graph convolutional neural networks may lead to different output feature matrices for the same state. This can result in excessive adjustments to certain dimensions of flight information and features during formation switching, hindering stable control of the flight formation.

[0089] Based on this, it is determined that the formation mode will be switched within a preset time period (e.g., 3 seconds) in the future, and the current first formation mode will be switched to the second formation mode. At this time, based on the first graph convolutional neural network corresponding to the first formation mode and the second graph convolutional neural network corresponding to the second formation mode, their respective first feature matrix and second feature matrix will be generated.

[0090] If there exists a flight feature of a specified dimension such that the difference between its value in the first feature matrix and the second feature matrix is ​​higher than a preset difference, for example, the velocity dimension in the feature matrix corresponds to an n×1 single-row parameter. If the difference between the values ​​of this parameter in the first feature matrix and the second feature matrix is ​​large, it indicates that different graph convolutional neural networks have large differences in their analysis results of the velocity feature.

[0091] At this point, based on the preset weights corresponding to the flight features of the specified dimension (generally speaking, different weights are set for different dimensions; for example, the weight of the current first feature matrix is ​​higher for state information such as speed, acceleration, altitude, heading angle, pitch angle, and roll angle, while the weights of the two are the same for external information such as wind speed and obstacle position, or the weight of the subsequent second feature matrix is ​​higher), the values ​​of the flight features of the specified dimension in the first and second feature matrices are weighted and summed to obtain the final value of the specified dimension.

[0092] In other words, for a given dimension with a large difference, a weighted summation method is used to reduce potential errors.

[0093] For the remaining flight characteristics, the difference between the first and second feature matrices is small, and can be considered negligible. Therefore, depending on whether the current moment is before or during a formation mode switch, the value from either the first or second feature matrix can be used as the final value. For example, if the current moment is before a formation mode switch, the value from the currently used first feature matrix is ​​used; if the current moment is during a formation mode switch, the value from the corresponding second feature matrix is ​​used. Based on the final value, the corresponding cooperative control commands for the UAV formation are generated and output.

[0094] Once the formation mode has been switched, the first feature matrix is ​​no longer used. Instead, the second feature matrix is ​​generated using the second convolutional neural network to generate the collaborative control commands.

[0095] As shown in Figure 4, this application also proposes a multi-drone cooperative control device based on a neural network, comprising:

[0096] At least one processor; and,

[0097] A memory that is communicatively connected to at least one processor; wherein,

[0098] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform, for example, the neural network-based multi-drone cooperative control method of any of the above embodiments.

[0099] This application also proposes a non-volatile readable storage medium storing computer-executable instructions, which are configured as: the neural network-based multi-drone cooperative control method for any of the above embodiments.

[0100] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0101] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0102] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more non-volatile readable storage media (including, but not limited to, disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code.

[0103] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0104] These computer program instructions may also be stored in a non-volatile readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the non-volatile readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0106] In a typical configuration, a computing device includes one or more processors (CPU, Central Processing Unit), input / output interfaces, network interfaces, and memory.

[0107] Memory may include non-persistent memory in non-volatile readable storage media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of a non-volatile readable storage medium.

[0108] Non-volatile readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of non-volatile readable storage media for computers include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, non-volatile readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0109] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0110] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for multi-drone cooperative control based on neural networks, characterized in that, include: Based on the current drone application scenarios, we determine the various formation methods required for drone formations, and for each formation method, we determine its corresponding topology. Based on the information flow direction between nodes in the topology, a corresponding topology matrix is ​​generated; Generate a graph convolutional neural network; the graph convolutional neural network includes at least a backbone network, a graph convolutional module, and an attention module; The skeleton network is used to extract multi-dimensional flight features corresponding to the UAV formation; The topology matrix is ​​used as the adjacency matrix, and the graph convolution module outputs the corresponding feature vector based on the multi-dimensional flight features and the adjacency matrix. The attention module outputs corresponding attention features based on the feature vector, and obtains the feature matrix corresponding to the UAV formation based on the attention features. Based on the feature matrix, the cooperative control commands corresponding to the UAV formation are output.

2. The method according to claim 1, characterized in that, Each of the formation methods corresponds to a topology; each of the topologies is trained individually to obtain its own corresponding graph convolutional neural network.

3. The method according to claim 1, characterized in that, The nodes in the topology represent unmanned aerial vehicles (UAVs), and the nodes are connected by directed edges. The direction of the directed edges is the information flow direction, which is the path and direction of information transmission from one node to another.

4. The method according to claim 1, characterized in that, Based on the information flow direction between nodes in the aforementioned topology, a corresponding topology matrix is ​​generated, specifically including: Determine the node number corresponding to each node in the topology; Select each node sequentially according to the numbering order corresponding to the node number, and assign each node to a single row parameter in the topology matrix; In each row of parameters, other nodes are selected sequentially according to the numbering order corresponding to the node number, except for the current node, and the corresponding parameters are generated based on the information flow direction between the current node and other nodes.

5. The method according to claim 1, characterized in that, The parameters of the graph convolutional neural network are optimized and adjusted for the key parameters of UAV flight. Multi-dimensional flight information is input into the graph convolutional neural network, and the graph convolutional neural network outputs the corresponding feature matrix. The multi-dimensional flight information includes the drone's speed, acceleration, altitude, heading angle, pitch angle, roll angle, wind speed, and obstacle positions.

6. The method according to claim 5, characterized in that, The multi-dimensional flight information is matched and mapped with the parameters of the graph convolutional neural network through model training; wherein, the training parameters are obtained by collecting historical data of each UAV formation and through preprocessing, annotation and other methods.

7. The method according to claim 1, characterized in that, The multi-dimensional flight features corresponding to the UAV formation are extracted through the skeleton network, specifically including: Based on the first fully connected layer of the skeleton network, the multi-dimensional flight information corresponding to the input UAV formation is matched with the shape of the input of the graph convolution module; The formula for the first fully connected layer is: y = W0 T x+b0; x is the input multi-dimensional flight information, y is the output multi-dimensional flight features, W0 is the weight matrix, and b0 is the bias matrix.

8. The method according to claim 7, characterized in that, The skeleton network is configured to perform input shape matching between the multi-dimensional flight information of the UAV and the graph convolutional neural network; the skeleton network is in the form of a ResNet network or a VGG network.

9. The method according to claim 7, characterized in that, The multi-dimensional flight features corresponding to the UAV formation are extracted through the skeleton network, specifically including: Set the size of the weight matrix W0 to (n-1)×n, the size of the bias matrix b0 to 1×n, and the size of the input multi-dimensional flight information x to n×n; where n is the number of dimensions of the multi-dimensional flight information. After multiplying the weight matrix W0 with the multi-dimensional flight information x, it is concatenated with the flipped bias matrix b0 to obtain the concatenated vector S; where the size of the concatenated vector S is n×n. Divide the concatenated vector S into multiple sub-vectors Si. i Each sub-vector is configured as one dimension of the corresponding multi-dimensional flight feature; where, sub-vector S i The size of the matrix is ​​n×1.

10. The method according to claim 9, characterized in that, The graph convolution module outputs a corresponding feature vector based on the multi-dimensional flight features and the adjacency matrix, specifically including: Each subvector S i The input is fed into the graph convolution module, and the output is the corresponding sub-feature vector T. i The sub-feature vector T i Concatenate the vectors to obtain the corresponding feature vector T; The formula for the graph convolution module is as follows: Among them, w t This is represented as the output of the graph convolution module, corresponding to the sub-feature vector T. i w s This is represented as the input to the graph convolution module, corresponding to the subvector S. i , Let ReLU be the adjacency matrix, and ReLU be the activation function. Wherein, parameter H (1) The formula is: Among them, w (l) Let b be the weight of the l-th layer. (l) For the bias of the l-th layer, H is the adjacency matrix, σ is the activation function; (l) The parameters of the graph convolution module represent the node features of the l-th layer. The initial value of l is 0. When l = 0 or 1, it corresponds to H respectively. (0) and H (1) .

11. The method according to claim 1, characterized in that, The attention module outputs corresponding attention features based on the feature vector, and obtains the feature matrix corresponding to the UAV formation based on the attention features, specifically including: The formula for the attention module is: M s =σ(f 5×5 ([AvgPool(F′);MaxPool(F′)])); Where σ is the Sigmoid function, f 5×5 For a 5×5 convolution process, M s Indicates attentional characteristics; [AvgPool(F′);MaxPool(F′)] is a channel concatenation of two feature matrices, AvgPool (average pooling) and MaxPool (max pooling). The parameters of both feature matrices are H×W×C, where H is the height of the feature matrix, W is the width of the feature matrix, and C is the number of channels of the feature matrix. The two feature matrices are concatenated by channels to obtain the concatenated feature parameters H×W×2C.

12. The method according to claim 11, characterized in that, The step of obtaining the feature matrix corresponding to the drone formation based on the attention features specifically includes: The attention features are input into the second fully connected layer, and the output is the feature matrix.

13. The method according to claim 1, characterized in that, After outputting the cooperative control command corresponding to the UAV formation based on the feature matrix, the method includes: One or more drones are designated as navigators, and the other drones are designated as followers. The navigator is responsible for planning the flight path and speed, and the followers adjust their flight status based on the information from the navigator.

14. The method according to claim 1, characterized in that, After outputting the cooperative control command corresponding to the UAV formation based on the feature matrix, the method includes: The movement of the drone is controlled in a distributed manner by setting behavioral rules for the drone.

15. The method according to claim 1, characterized in that, After outputting the cooperative control command corresponding to the UAV formation based on the feature matrix, the method includes: The drone formation is guided to fly by a virtual navigation point.

16. The method according to claim 1, characterized in that, After outputting the cooperative control command corresponding to the UAV formation based on the feature matrix, the method includes: Multiple drones are virtualized into a single structure, and the drone formation is controlled by controlling this structure.

17. The method according to claim 1, characterized in that, For each formation method, determine its corresponding topology, specifically including: For each formation method, determine the corresponding number of drones, as well as the corresponding center node and edge node in the formation method; If the number of drones is lower than a first preset threshold, then a preset number of nodes are selected from the edge nodes as navigation nodes so that the out-degree value of the navigation nodes in the topology is the highest. If the number of drones is higher than a first preset threshold, then the difference between the out-degree values ​​of all nodes in the formation is made lower than a second preset threshold.

18. The method according to claim 1, characterized in that, After obtaining the feature matrix corresponding to the drone formation based on the attention features, the method further includes: The formation mode will be switched within a preset time period in the future, changing the current first formation mode to the second formation mode; Based on the first graph convolutional neural network corresponding to the first formation method and the second graph convolutional neural network corresponding to the second formation method, respectively generate their respective first feature matrix and second feature matrix; If there exists a flight feature of a specified dimension such that the difference between it and the first feature matrix and the second feature matrix is ​​higher than a preset difference, then based on the preset weight corresponding to the flight feature of the specified dimension, the values ​​of the flight feature of the specified dimension in the first feature matrix and the second feature matrix are weighted and summed to obtain the final value of the specified dimension. For the remaining dimensions of flight characteristics, depending on whether the current moment is before or during the formation mode switch, the value in the first feature matrix or the second feature matrix is ​​selected as the final value. Based on the final value, the cooperative control command corresponding to the UAV formation is generated and output.

19. A multi-drone collaborative control device for unmanned aerial vehicles (UAVs) based on neural networks, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform, for example, the neural network-based multi-drone cooperative control method as described in any one of claims 1 to 8.

20. A non-volatile readable storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: the neural network-based multi-drone cooperative control method for unmanned aerial vehicles as described in any one of claims 1 to 18.

Images