Methods, systems, and computer program products for selecting a target

Abstract

A computer-implemented method of selecting a target for an engagement system, the method comprising: determining at least one engagement means comprised in the engagement system, wherein the engagement means is configurable to engage one or more targets; determining a plurality of target representations, wherein each target representation is indicative of a position, and preferably movement, of one of the targets relative to the engagement means; determining, by an agent component comprised in the engagement system, a context representation indicative of a current and / or past situational context; comparing, by the agent component, each of the target representations to the context representation to determine a respective score for each of the targets; determining, based on the scores, a target selection for the engagement means; and transmitting the target selection to the engagement means.

Description

Methods, Systems, and Computer Program Products for Selecting a TargetTechnical Field

[0001] The present disclosure relates to systems, methods, and computer program products for selecting a target for an engagement system. The disclosure is applicable in the field of threat evaluation and weapons assignment, in particular in view of applications in aerospace defence, in particular air defence.Background

[0002] A frequent problem in air defence is defending an area against attack by multiple targets, such as aircraft, e.g. unmanned aerial vehicles. This is in particular relevant as the targets may pose themselves a threat to an air defence system and / or protected assets, such as buildings, vehicles, aircraft, or ships. Therefore, operating an air defence system requires selecting the targets to engage in a way that leads to successful air defence. In particular, quick decisions must be made on which target is to be engaged at which time.

[0003] Known air defence systems are manually operated. Human threat assessment and decision-making can be so slow that only a limited number of targets can be destroyed despite a sufficient number of weapons being available. Known automatic systems, such as radar-controlled anti-aircraft artillery, provide only limited capabilities regarding target selection.

[0004] There is a need for systems and methods that overcome these shortcomings.Summary

[0005] Disclosed and claimed herein are systems, methods, and devices for selecting a target.

[0006] A first aspect of the present disclosure relates to a computer-implemented method of selecting a target for an engagement system. The method comprises the following steps:• determining at least one engagement means comprised in the engagement system, wherein the engagement means is configurable to engage one or more targets;• determining a plurality of target representations, wherein each target representation is indicative of a position, and preferably movement, of one of the targets relative to the engagement means;• determining, by an agent component comprised in the engagement system, a context representation indicative of a current and / or past situational context based on target information on one or more of the targets, engagement means information on one or more of the engagement means, and / or protected assets information on one or more protected assets, preferably based on indications of positions of the one or more targets, a status of one or more of the engagement means, and more preferably also one or more protected assets;• comparing, by the agent component, each of the target representations to the context representation to determine a respective score for each of the targets, in particular wherein the score is indicative of a similarity of the respective target representation to the context representation;• determining, based on the scores, a target selection for the engagement means; and • transmitting the target selection to the engagement means.

[0007] An advantage of this method is that the target selection is based both on target representations and a situational context, e.g. status of the engagement means. This allows taking into account the context, in particular present and / or past observations on targets, engagement means, and or protected assets to be protected by the engagement system. The observations preferably include locations of targets, and more preferably their velocities. The observations may also include type, status, and / or amount of ammunition, of targets, engagement means, and / or protected assets. More particularly, observations on past and ongoing engagements can be taken into account for the determination of the context representation.

[0008] Moreover, the target selection can be interpreted as a prioritised target list. That is, a human-understandable output of target priority scores is determined. Moreover, the method is adaptable to downstream algorithms that may determine, e.g. an engagement means selection based on the priority scores.

[0009] The context representation and the target representations are preferably vectors.Comparing the target representations to the context representation may comprise performing a matrix multiplication. The matrix multiplication is preferably conducted by stacking the target representations to a matrix and multiplying the matrix with the context representation.

[0010] The engagement means may be configured to engage a target in response to receiving the target selection.

[0011] In embodiments, the engagement means may be configurable to engage at most one target of the one or more targets at a time. This may be the case for artillery pieces or radiation weapons that can only engage one target of the one or more targets at a time. The agent component is preferably adapted to this kind of engagement means, such that only computing is conducted that is necessary to control this kind of engagement means. In this case, it may be possible to control physical weapons systems that can engage a plurality of targets at a time .

[0012] Higher scores may be indicative of a higher priority of engaging the target. In particular, higher scores may be indicative of, or related to, engaging the target being a part of a successful policy, e.g. a successful strategy comprising engaging the target at the given instant.

[0013] Higher scores may be indicative of a higher similarity of the corresponding target representation to the context representation. This way, the context representation determined by an appropriately trained agent component is similar to the target representation related to the target that should be engaged to arrive at a successful strategy that maximizes the rewards.

[0014] Determining the target selection may comprise selecting a target related to a maximum score. This provides an advantageously simple way of selecting a target in case the engagement means can only engage a target at a time. In particular, the selection of a target related to a maximum score, or one of the highest scores, has the effect that the algorithm that generates the context representation can be trained to generate, in those cases where selection of a given target is most conducive to a successful strategy, a context representation most similar to the representation of the target.

[0015] In embodiments, determining the target selection comprises selecting one or more targets related to the highest scores. This offers a way of determining targets for an engagement that is operable to engage a plurality of targets. In particular, determining the target selection comprising the target with the highest score. This way, a target may be determined for an engagement means that is operable to engage a target at a time. In inference, this enables selection of targets corresponding to the strategy applied by the agent component in the present environment.

[0016] In an embodiment, the target representations comprise a no-operation representation indicative of an instruction not to engage any target. This allows developing a strategy in which it is an advantageous or even frequent decision to conduct no engagement.

[0017] In an embodiment, the context representation is determined by a recurrent neural network, RNN, preferably a long short term memory, LSTM, comprised in the agent component based on a situational awareness representation based on positions, and preferably movement, of targets and / or engagement means. This allows determining the context representation as a representation of a state of the machine learning algorithm in the reinforcement learning framework. The situational awareness representation may be supplied to an input of the RNN directly or after processing by a feedforward neural network, such as an MLP.

[0018] In an embodiment, the or a situational awareness representation is determined by processing one or more of the target representations and, preferably, one or more engagement means representations, and more preferably, also one or more protected asset representations. This has the advantage that the target representations are, once determined, used for two computing steps, i.e.,• to determine the context representations, and• to compare the context representation to the target representations.

[0019] This has the advantage that computing steps to determine the target representations are used for the two downstream processing steps, so that compute power is used more efficiently.

[0020] In an embodiment, the or a situational awareness representation comprises a mean embedding of the target representations and one or more engagement means representations indicative of a status of each of the engagement means. Put differently, the situational awareness representation may comprise an average of one or more target representations and one or more engagement means representations. The situational awareness representation may further comprise other data, such as information on protected assets. In case of a large number of targets, the target representations may comprise a large amount of data. Therefore, calculating a mean embedding provides a way of lossy compression of the target and engagement means representations. Therefore, the amount of data supplied to and processed by the RNN is reduced. This reduces the computational load generated by the particularly computationally intensive recurrent neural network.

[0021] In an embodiment, the or a situational awareness representation comprises, and preferably is, a concatenation of the mean embedding and an asset representation of properties of the one or more protected assets. This has the advantage that the asset representation is emphasized compared to the information on the targets and engagement means. The RNN therefore processes more data depending on the protected assets, which causes it to take the protected asset more into account. It has been found out that the agent component trains better if the concatenation is used.

[0022] In an embodiment, the target representations and / or the one or more engagement means representations are determined by a backbone machine learning algorithm based on input representations, the backbone machine learning algorithm preferably comprising a selfattention based backbone neural network, more preferably an encoder of a transformer-type neural network.

[0023] The backbone-generated representations for the engagement means and targets are better suitable for processing by both by the RNN and for the comparison to the context vector in the output heads. Moreover, the use of a transformer-type network in particular has been found to learn the generation of more suitable representations via the self-attention mechanism, and learns implicit relations between the targets and engagement means. The backbone network is preferably an encoder of a transformer-type network. Such networks can be run at sufficient speed on available hardware to allow processing, by the entire agent component, the observation up to the determination of the target selection within less than 500 milliseconds.

[0024] In an embodiment, the input representations are based on, preferably comprise, sensor data indicative of:• positions, and preferably movement, of the one or more targets; and / or• a status of the engagement means.

[0025] A second aspect of the present disclosure relates to a computer-implemented method of training an agent component, in particular an agent component comprising a machine learning algorithm of any of the preceding claims, wherein the agent component pertains to the or an engagement system comprising a plurality of engagement means configurable to engage one or more targets. The method comprises the following steps:• determining training data comprising observations of an environment indicative of a position, and preferably movement, of the one or more targets relative to the engagement means;• determining, by the agent component and based on the training data, a target selection for each of the engagement means; and / or• updating the agent component based on single-agent reinforcement learning.

[0026] This process, reinforcement learning, has been found to allow training of the agent component to develop a successful engagement strategy. The process may comprise determining a reward based on• a determination that one or more engagements of one or more selected targets by one or more engagement means are successful;• a determination no more targets are found in the environment; and / or• a determination that high-value asset has been destroyed.

[0027] The agent component preferably controls a plurality of engagement means. Thereby, one agent component may be trained to control a plurality of engagement means, such that it can develop a strategy of operating all engagement means to obtain a high reward. The agent component more preferably comprises a plurality of target selection algorithms as output heads, wherein each of the target selection heads is configured to determine an action command for each of the engagement means. This allows the agent component to be trained without resorting to a more complex multi-agent learning process. The agent component more preferably comprises a recurrent neural network (RNN). The output of the RNN may be coupled to an input of each of the target selection algorithms. Training the agent component may comprise training the RNN by single agent reinforcement learning to control the target selection algorithms based on the above-mentioned reward.

[0028] In embodiments, determining a target selection comprises stochastically selecting one or more of the one or more targets.

[0029] This means that training may comprise setting the weights for the stochastically selected targets, which has been shown to lead to convergence in realistic scenarios. Training may comprise training by Proximal Policy Optimization (PPO).

[0030] During the training the agent component may execute any of the steps of the method of the first aspect. In particular, the observations may comprise training target representations, and determining a target selection may comprise:• determining a training context representation indicative of a current and / or past situational context based on the observations;• comparing each of the training target representations to the training context representation to determine a respective score for each of the targets;• determining the target selection based on the scores wherein higher scores are preferably indicative of a higher similarity of the corresponding training target representation to the training context representation; and• selecting one or more targets related to the highest scores, in particular selecting a target related to the highest score; and / or stochastically selecting one or more targets.

[0031] The observations may comprise: target information on one or more of the targets, engagement means information on one or more of the engagement means, and / or protected assets information on one or more protected assets, preferably based on indications of positions of the one or more targets, a status of one or more of the engagement means, and more preferably also one or more protected assets.

[0032] In particular, the score may be indicative of a similarity of the respective target representation to the context representation.

[0033] The agent component may be comprised in the engagement system.

[0034] In embodiments, the method comprises attributing a first positive award in response to a determination that an engagement is successful, preferably during training immediately after a time step at which successful engagement was determined. The method may comprise attributing, preferably at the end, a second positive reward if no more targets are found in the environment, wherein the second positive reward is preferably larger than the first positive reward. The method may comprise attributing a negative reward in response to a determination that a high-value asset has been destroyed. A high-value asset may relate to any object that the agent component is to be trained to protect.

[0035] In embodiments, the self-attention based machine learning algorithm forms part of a reinforcement learning based agent component, and the agent component preferably comprises:• the or a backbone machine learning algorithm configured to determine the target representations and engagement means representations;• the or a recurrent neural network configured to determine the context representation based on the target representations and engagement means representations; and / or • at least one, preferably a plurality of, the target selection algorithms as output heads, wherein each of the target selection heads is configured to determine an action command for one of the engagement means.

[0036] The recurrent neural network is preferably further configured to determine the context representation based on the target representations, the engagement meansrepresentations, and one or more asset representations indicative of protected assets. The protected assets comprise objects the agent component is configured to protect. They preferably comprise the high-value targets. Thereby, the agent component may be trained to develop a strategy based on properties of the protected assets in a way that avoids loss of these assets.

[0037] In embodiments, the agent component of the method of the first aspect may have been trained according to the method of the second aspect. A third aspect of the present disclosure relates to a system comprising one or more processors and one or more storage devices, wherein the system is configured to perform the computer-implemented method of the first and / or second aspect of the present disclosure.

[0038] A fourth aspect of the present disclosure relates to a computer program product for loading into a memory of the system. The computer program comprises instructions, that, when executed by a processor of the system, cause the system to execute a method of the first and / or second aspect of the present disclosure.

[0039] In an embodiment, a non-transitory computer-readable storage medium is provided that stores instructions executable by one or more processors. The instructions comprise any of the steps of a method of the first and / or second aspect of the present disclosure.

[0040] A fifth aspect of the present disclosure relates to a computer program product comprising a trained machine learning module obtainable by the method of the second aspect of the present disclosure. In an embodiment, a non-transitory computer-readable storage medium is provided that stores the trained machine learning module.Brief description of the drawings

[0041] The features, objects, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference numerals refer to similar elements.• Fig. l is a flow chart of a method of controlling an engagement system;• Fig. 2 is a flow chart of a method of providing an agent component of an engagement system;• Fig. 3 is a block diagram showing a structure of an agent component and corresponding data types;• Fig. 4 is a block diagram showing a structure of an engagement means head and corresponding data types; and• Fig. 5 is a schematic drawing of a scenario.Detailed description of the preferred embodiments

[0042] Fig. 1 is a flow chart of a method 100 of selecting one or more targets for an engagement means of an engagement system, such as engagement means 512, 514, of engagement system 510 shown in Fig. 5.

[0043] The steps of method 100 may be executed by an agent component, in particular an agent component obtainable by method 200 shown in Fig. 2, comprising structures as shown in Fig. 3 and / or Fig. 5, and / or the agent component 520 of Fig. 5. Method 100 allows operating the engagement system to engage, preferably destroy, one or more targets.Generally, the agent component processes the data during training, in particular in step 206 (determining an action) in a similar or identical manner as in inference. This may include executing the steps of method 100, possibly subject to some differences that are pointed out below.

[0044] The engagement system may comprise any system operable to engage one or more targets. The method allows engaging the targets as part of a combat strategy. Some or all of the targets themselves may include a threat, i.e. be configured to engage parts of the engagement system and / or one or more protected assets that the engagement system may be designed to protect. In this case, the strategy may be developed such that the engagement system aims at successful engagement of targets before the targets themselves can successfully engage the engagement system and / or protected assets.

[0045] The engagement system is preferably trained by a reinforcement learning method, such as method 200 shown in Fig. 2.

[0046] The method is preferably conducted in a loop, so that the steps are repeated. The steps may be repeated at a predetermined frequency, e.g. once every 500 milliseconds. Each loop iteration thus determines a time step. The steps may be repeated during varying time intervals determined by processing time. The steps may have a fixed duration. Fast processing allows quickly reacting to any change observed by the sensors.

[0047] Method 100 begins by determining, 102, one or more engagement means. The engagement means are comprised in an engagement system. Each of the engagement meansis configurable to engage one or more targets. In embodiments, each of the engagement means may be configured to engage with one target at a time. Step 102 preferably includes determining for each of the available engagement means an alive status that indicates if the engagement means is functioning, and an ammunition status that indicates the amount of ammunition left. Based on the determination of the engagement means, an engagement means representation may be determined that is preferably based on the alive status and the ammunition status.

[0048] At 104, target representations are determined. The target representations may include any representation of a position, and preferably also movement, of a target. The target representations are preferably determined based on sensor data, such as sensor data obtainable by a sensor operable to determine a target position. It is preferred to process the sensor data by a machine learning algorithm to determine the target representations. It is further preferred to determine the target representations by a backbone neural network that processes both the sensor data and information on the engagement means, such as the alive and / or ammunition status to determine both target representations and engagement means representations. Examples of a backbone neural network are described with reference to Fig.3.

[0049] At 106, context representations are determined. The context representations are based on an observation, preferably comprising the sensor data indicative of the targets and the status of the engagement means. It is further preferred if the context representations comprise a status of a protected asset.

[0050] The information on the engagement means, the target, and the optional information on protected assets can be seen as an observation of an environment in which the engagement system acts.

[0051] The context representations may be determined, 108, by a recurrent neural network, RNN. This has the advantage that the context representations are based not only on input data (engagement means status, target data, and preferably data on protected assets) at the current iteration of the loop but also data determined during past iterations. The temporal development is thereby taken into account.

[0052] This means that a velocity of one or more threats may be determined implicitly by the agent component, even if no velocity data is input into the agent component. In other words, the target representation for a target may comprise information about the velocity of the target, although this information is generally not human-readable. Similarly, the contextrepresentation may generally depend, i.e. be based on the trajectories and velocities of the targets, even if this information is not encoded in human-readable form in the context representation. Rather, the context representation can be seen as a result of a machinelearning based reasoning on how to act, by the engagement system, in the environment determined by the observation.

[0053] The context representations may be determined based on the target representations already determined in step 104. In particular, the target representations as determined by one and the same previous calculation steps may have two purposes:• It serves as an input to the recurrent neural network for obtaining the context representation, and• the context representation is compared to the target representation.

[0054] At 110, the target and the context representations are compared. In particular, each target representation is compared to a context representation to determine a corresponding score. The comparison preferably comprises the determination of a product of each of the target and context representations. The determination of the score has the advantage that it is human-readable, i.e. the score is a measure of the degree to which the engagement of the corresponding target at the present loop iteration is conducive to a successful strategy.

[0055] At 112, the agent component determines a target selection. It is preferred to determine a target selection based on the score, in particular by selecting the target with a maximum score, 114. In case multiple targets can be engaged, the targets with the highest scores may be selected. However, in embodiments, the target may be selected at least in part stochastically, e.g. by determining a subset of targets having a score higher than a predetermined threshold, or by determining a subset of a predetermined number of targets with the highest scores, and subsequently selecting targets in the subset randomly. In preferred embodiments, the agent component is configured to select the target with the maximum score during inference, and to stochastically sample targets during a training phase. Stochastically sampling may be part of a Proximal Policy Optimization (PPO) approach.

[0056] The target selection may comprise a selection of conducting no engagement of any target at the present time step and / or loop iteration. An agent component configured to make such a selection can develop a strategy in which during one, or preferably many time steps, no action is taken. This has the advantage that even at a high number of loop iterations per time unit, which enables fast processing, the engagement system can take no action for asizable time interval. This allows developing strategies in which the engagement system can wait, e.g., for targets to approach until they are in range of the engagement means.

[0057] At 118, the target selection is transmitted to an engagement means. The transmission may cause the engagement system to engage a target. Transmission of the target selection may include an evaluation step, wherein it is first determined if a delay command for all engagement means is comprised in the action command. In this case, no action commands are transmitted. It is further preferable to obtain, 120, a confirmation user input. This may be realized as a human-in-the-loop solution, where the action command is output to a human interface device to seek confirmation from the user that the action commands can be transmitted, and wherein the action commands are transmitted only if an affirmative user input is received. This may, however, also be realized as a human-on-the-loop solution, wherein the action commands are transmitted to the engagement means if a user input enabling transmission has been received within a predetermined time interval, which is preferably much longer than the duration of execution of steps 102 to 118. This is preferable since the user input can be received asynchronously, i.e. the user may authorize autonomous operation of the engagement system in advance for an interval of, e.g. 15 minutes.

[0058] By the method 100, the targets are selected, according to a machine-learning based threat assessment. In a real -world application, the method 100 is based on sensor data and controls an engagement system. The data may, however, also comprise synthetic data, such as data from a simulation. Method 100 allows increasing kill probabilities and reducing the risk of loss of engagement means and / or sensors and / or high-value assets, and reducing the requirements for ammunition.

[0059] Method 100 is preferably applicable for defending, by the engagement system, one or more protected assets against a saturation attack where a large number of targets appear at about the same time.

[0060] Fig. 2 is a flow chart of a method 200 of providing an agent component of an engagement system.

[0061] Method 200 may be seen as a reinforcement learning method. According to the concept of reinforcement learning and subject to the specifics explained in this description, an agent component is generally configured to observe a real or simulated environment and to act in the environment by determining actions. In particular, observations, i.e. data determined based on and / or indicative of the environment, may be supplied to an input of the agent component. It is preferable to process observations by preprocessing or by othermachine learning algorithms, such as feedforward neural networks, to generate a representation that is then input into the agent component. The agent component is configured to process the observations and / or representations and to determine a target selection as an output, which may be seen as an action of the agent component on the environment in the reinforcement learning context. The action may influence the environment, e.g. by being input into a controlled device, e.g. an engagement means.Training may require large amounts of training data to generate a robust model that can be used for inference on testing data. During training, intermediate and / or final rewards may be applied, and the agent component may be trained to increase, preferably maximize the rewards as detailed below.

[0062] The method 200 begins by initializing, 202, the machine learning algorithm. This preferably includes configuring the agent component such that it generates a random action in response to an observation. Then, the subsequent training steps determine the agent component's learned behaviour.

[0063] The agent component is trained in this example by interacting with a simulated environment. The simulation preferably includes one or more targets, such as attacking UAVs. The simulation preferably further comprises the engagement means and sensors of the engagement system, and more preferably, one or more protected assets. The simulation may comprise a large number of targets arriving at the same time, destruction of sensors and / or engagement means due to engagements conducted by the targets, and other changes determined by the simulation setup. The simulation comprises also the actions taken by agent component, in particular in reaction to the outputs generated, i.e. decisions made, by the agent component. The simulation preferably runs in a headless mode during training, where no graphics are rendered as far as they are not necessary to generate observation data for input into the agent component. The interaction between the simulation and the processing of the observations by the agent component is conducted as follows: The steps 204 - 212 are repeated for each time step, which preferably has a predetermined duration. During each time step, the simulation environment processes the simulation to reflect both the changes imposed by the simulation setup, e.g. the appearance of new targets, and the changes caused by the actions determined by the agent component, such as successful engagements of threats. Moreover, during each time step, observations that can be made by the agent component are determined and processed to generate an action in response to the ongoing simulated situation, which is then fed into the simulation on the next time step.

[0064] At 204, the observations are supplied to the agent component. The observations may be determined by a simulation. It is preferred to use a simulation environment that captures a scenario on which the agent component is to be trained. In order to allow the agent component to generalize, it is further preferred to conduct a plurality of simulations and to supply the observations generated by the simulations as a batch of inputs to the agent component. The simulations progress for some time to determine a new simulation state, and observations are determined from the state by extracting, from the simulation, the states of targets that are observable using the sensors of the simulation, and further extracting the corresponding engagement states, engagement states, and sensor alive status information. This preferably includes updating of an alive status one or more sensors and engagement means, i.e. the alive status is set to False if the simulation has determined the corresponding sensor or engagement means to be destroyed during the present time step. This preferably includes updating the ammunition status of the engagement means to reflect ammunition expended during the last engagement step.

[0065] At 206, the agent component determines an action to be conducted in response to the observation. This preferably includes processing the observation by a neural network as shown in Fig. 3 and / or Fig. 5. The agent component may execute all steps that may be executed during inference, and is, in addition, trained to develop an engagement strategy. During training by Proximal Policy Optimization, the targets are selected stochastically. Preferably, a backbone layer generates a representation of the observation, and a recurrent neural network then generates outputs that include an action command for each engagement means. Determining the action command includes selecting at least one target for the engagement means as described with reference to Fig. 1. It is preferred to select the target by random selection 116, which allows for faster convergence.

[0066] Each such action command may include a no-operation command that causes the corresponding engagement means not to engage the target at the present time step. The action command may include an engage command comprising an indication of which target to engage. Since the same agent component determines the action commands for all engagement means, the behaviour of the engagement means is inherently coordinated. This has the advantage that the agent component can be trained to determine an action by the entire engagement system, which may include attacking a high-value target with a plurality of engagement means at the same time, and attacking a low-value target only with one engagement means or none at all.

[0067] At 208, the action is applied to the environment. This preferably includes outputting, by the agent component, the action commands comprising the target selection to simulated engagement means, so that the agent component acts on the environment at this step.

[0068] At 210, an intermediate reward is determined that depends on the result of the action command generated during the last time step and / or preceding steps. More specifically, the intermediate reward may depend on the history of the evolution of the scenario. If the simulation has determined that an engagement of a target has led to the target being destroyed, a positive reward is attributed. Determining the rewards in this way allows training the agent component to increase, preferably maximize, the kill probabilities reached by the engagement system. The magnitude of the reward preferably depends on the importance of the target. In particular, the simulation setup may comprise a user-determined indication of the importance of each target. The indication is then used to determine the reward during the time step. Moreover, it is preferred to apply a negative reward for any ammunition used. The magnitude of the negative reward depends on the type and scarcity of the ammunition. It is preferred to set predetermined parameters to determine the type of the ammunition as part of the simulation setup. The scarcity of the ammunition may be determined as a function of the amount of ammunition left. It should be noted, however, that the reward does not necessarily depend directly on the target selection. The agent component is preferably trained based on reward functions that depend on the outcome of the target selection as it is determined in the simulated environment. This allows taking into account effects that are part of a complex simulation. For example, a decision to engage a particular target, e.g. a high value target, may appear as most advantageous in a given situation, but the simulation may determine that the engagement is unsuccessful. In this case, no positive reward should be given, i.e. it is advantageous if the reward is determined based on the outcome, i.e. whether the target is destroyed.

[0069] At 212, it is determined if an abort condition has been reached. An abort condition may include the determination that there are no more targets left (successful outcome) or that all engagement means and / or all sensors and / or all protected assets have been destroyed. If no abort condition is fulfilled a time step is determined at 212. This preferably comprises incrementing a current time value by a predetermined value, which is preferably at about 500 milliseconds. This value allows training the agent component to develop a temporally finegrained coordination strategy, that comprises reacting to any changes in the situation quickly. After the time step, the next iteration of steps 204-210 is conducted.

[0070] If an abort condition is reached, a terminal reward is applied, 214. This includes a large positive reward if there are no more targets left, and a large negative reward if a high-value asset is destroyed. The terminal reward is zero otherwise.

[0071] By method 200, the agent component has learned a policy, which is a mapping from observations to actions. Put differently, the setting of the weights in the agent component enables the trained agent component to determine an action for any test observations that could be supplied to an input of the agent component. The determination of the action is based on the training process and generally causes it to generate actions that lead to a high reward. It should be noted that the agent component itself can be seen as a model-free agent, since the model of the situation is comprised in the simulation, which can be a separate algorithm that communicates with the agent component only by determining the states and observations, and by determining the reward. The training of the agent component based on the intermediate rewards and the terminal reward can be done using Proximal Policy Optimization (PPO) as explained, e.g., in: Schulman et al.: Proximal Policy Optimization Algorithms. arXiv: 1707.06347v2 [cs.LG] (2017).

[0072] In an exemplary training process, the agent component has been trained using 80 instances of a simulation environment in a headless mode. For each iteration of training, 80 different runs of the scenario were used as a batch. Each episode was up to three minutes long, and the agent component could act in discrete steps for at most 300 steps.

[0073] Fig. 3 is a block diagram showing a structure of an agent component 320 and corresponding data types. The agent component may be implemented as a software run on a computing system comprising processors and memory devices used as a component of a command control centre of an engagement system. The agent component 320 comprises inputs to receive input data pertaining to an observation 300. The observation 300 comprises a target status 302 indicative of a status of one or more observed targets. In an application, such as the scenario shown in Figure 5, not all targets are necessarily observed. However, the approach of the present disclosure allows determination also in the case of a partial observation reinforcement learning problem. The target status comprises two target input representations 304 and 306. Each target input representation comprises an indication of a position of the target. It may optionally also comprise an indication of a velocity of the target. It may optionally also comprise an indication of a type of the target. The optional data enables the agent component to develop strategies that take into account to which extent a target poses a threat. For example, a classification as a certain target type, or a determinationof a fast-moving target may result in a treatment of the target as a high-value target.However, the target type is optional since the model can also learn an according strategy if only target positions indicated in the target input representations 304, 306. This is in part due to the use of a recurrent neural network as described below. Since the target type is not known in many real world scenarios (e.g. not detected by any sensor, which may include a radar device), it is preferred not to use a target type as input and instead to rely on the capability of the agent component to learn a strategy without receiving an input indicative of the target type.

[0074] The observation 300 further comprises an engagement means status 308 comprising engagement means representations 310, 312, each of which is related to an engagement means comprised in the engagement system. The engagement means representations 310, 312 preferably comprise a status, in particular a status indicative of whether the engagement means is in operation, and / or a status indicative of an amount of available ammunition of the corresponding engagement means.

[0075] The observation 300 preferably further comprises a protected assets status 314comprising protected assets representations 316, 318. Each of the protected assets representations 316, 318, is indicative of a protected asset that the engagement system is configured to protect. A protected asset may be any object, e.g. a building, a vehicle, an aircraft and / or a vessel. This allows training the agent component to develop strategies that prevent successful counter-engagement of the protected assets related to the protected assets representations 316, 318, e.g. by engaging targets before they can engage the protected assets. The protected assets representations 316, 318preferably include a position, and preferably also a velocity and / or type of the protected asset, which allows developing a strategy to protect preferably high-value protected assets.

[0076] The agent component 320 may comprise one or more feed-forward neural networks, in this case multilayer perceptrons 322, 324, 326, 328, 330, 332 to generate a representation of each of the input representations. The representations of target and engagement means are then preferably input into a backbone layer 334.

[0077] The backbone layer 334 is preferably a self-attention based neural network, more preferably an encoder part of a Transformer type network. Transformer-type neural networks are described in, e.g., Vaswani et al: Attention Is All You Need. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. The Transformer network has the advantage that the encoder type is inherently parallelizable. The targetrepresentations 338, 340 and the engagement means representations 342, 344 have the same structure as the input. However, each data point therein is based on the entire data input, i.e. target status 302 and engagement means status 308. The backbone layer has a fixed input size that determines a maximum number of inputs, i.e. the sum of the numbers of targets and engagement means, it can process. If the number of targets in the environment that are currently observed is lower than the maximum, the remaining inputs are supplied with dummy inputs. The maximum input size may be quite high because the encoder layer is inherently parallelizable and is therefore computationally efficient.

[0078] The target representation dataset 336 consists of target representations 338, 340, each of which is related to a respective target and based on the respective target input representation. Each of the engagement means representations 342, 344 is related to, and based on, a respective engagement means. In embodiments, the target representations 338, 340 and engagement means representations 342, 344 are determined based on all inputs of the backbone layer 334, including target and engagement means input representations.

[0079] A mean is calculated, at 346, of the target representations 338, 340 and the engagement means representations 342, 344. The mean calculation 346 has the advantage that it reduces the amount of data to be processed by the more computationally demanding recurrent neural network 356. The mean function is calculated as a mean over the target representations 338, 340 and the engagement means representations 342, 344. and results in a mean embedding 348.

[0080] The mean embedding 348 may have a data format of a vector of a same length as each of the target representations 338, 340 and engagement means representations 342, 344.

[0081] The mean embedding 348 is concatenated, 350, with protected asset representations 316, 318. The result of this steps is referred to as situational awareness 352 as it comprises information on targets, engagement means, and assets. It is preferred to process each of the protected asset representations 316, 318, by a feed-forward neural network, such as MLPs 330, 332, to generate representations to be concatenated with the mean embodiment 348. The concatenation has the effect that the information on the protected assets is amplified compared to the information on targets and engagement means. In other words, a part of the information on the targets and engagement means is lost, whereas more information on the protected assets is emphasized by occupying a larger number of bits in the situational awareness 352. The situational awareness 352 is preferably processed by a feed-forward neural network, such as MLP 354, prior to being input in a recurrent network (RNN) 356.The recurrent neural network may, e.g., comprise a long-short term memory (LSTM), such as the LSTM described in: Hochreiter and Schmidhuber: Long short-term memory. Neural computation 9, 1735 (1997). The RNN 356 generates an output that is based on current and past inputs to the RNN. This enables the agent component to develop a strategy wherein each step is not only based on current inputs, but also past inputs. For example, a behavioural pattern of a target over time may lead to a target being processed as a high-value target in inference if, e.g. preferred engagement of similar targets was positively rewarded during training. The agent component can moreover develop strategies that depend on a velocity of a target due to the recurrent nature of the neural network 356, even if no velocity is included in target representations 302.

[0082] The output of the RNN 356 may be referred to as a context representation, since it comprises information on the context, i.e. information derived from past and present development of the observation 300. The output of the RNN 356 is then processed by an output layer 358. The output layer may comprise one or more engagement means heads 360, 362, each of which is communicatively couplable to an engagement means. In particular, an output of each of the engagement means heads 360, 362, may be coupled to an input of the engagement means. It is preferred to supply a copy of the output of the RNN 356 to each of the engagement means heads 360, 362. Each of the engagement means heads is further preferred to receive a copy of the target representation dataset 336. This enables the engagement means head to determine the action command for the corresponding engagement means by way of comparison of representations as shown in Figure 4.

[0083] It should be noted that at every time step, the agent component 320 receives the observation 300, processes it by generating the data structures shown in Fig. 3, and generates the target selection 364 for that time step. The components and data structures upstream of the RNN 356, such as the situational awareness 352 do only comprise information based on the observation 300 received at that time step. However, the RNN 356 stores information on the observation 300 received at previous time steps. It is therefore the RNN that enables learning a strategy that takes into account the development of the observed environment. The context representation output by the RNN 356, and the target selection, are therefore based on the past development of the observed scenario.

[0084] Fig. 4 is a block diagram showing a structure of an engagement means head 406 and corresponding data types. The engagement means head 406 may be an implementation of one or more, in particular each, of engagement mean heads 358, 360 shown in Figure 3. Theengagement means head 406 has an input to receive target representation dataset 400, which may be an implementation of target representation dataset 334 shown in Figure 3.

[0085] The target representation dataset comprises target representations T 1 ' 402 and Tn' 404. The target representations 402, 404 are preferably embeddings of features of the target and may be represented as a vector data format. Put differently, the target representations are based on one or more properties of the target, including the position, and may encode this property by constituting an element in a vector space. Differences between representations generally represent a degree of differences of the features of the target. The features preferably include, in addition to a position, also a velocity and / or target type. Targets representations 402, 404 having a low distance to one another may denote targets that are physically close to one another, have similar or identical velocities, and / or are of similar or the same type. However, the target representations 402, 404 need not comprise a human-understandable or trivial encoding of any of the properties. Determination of each of the target representations 402, 404, may involve processing features of other targets as described with respect to Figure 3.

[0086] The context representation 406 is based on target and engagement means. This means that the context vector comprises information, which may not be human-readable, on targets and engagement means, and preferably also protected assets. It is preferred if the context representation is also based on a past temporal development of the targets, engagement means, and / or protected assets. The context representation 406 may be obtainable by a recurrent neural network, such as RNN 356 of Figure 3.

[0087] The engagement means head preferably comprises a feed-forward neural network, which in this case is a multilayer perceptron 410. It is preferred to use a fully connected multilayer perceptron. The feed-forward neural network is configured to receive as an input the target representation dataset 400 and to determine processed target representations 412, 414. The processed target representations 412, 414, are situated in the same vector space and have a same dimensionality as the context representation 406. The context representation 406 may be seen as containing information on which target selection would be most conducive to an overall successful policy. A difference between a target representation and the context representation 406 may be understood as a difference between the respective target properties and the properties of a target that should be ideally selected. It is further preferred to define a no-operation representation 416 indicative of a decision not to engage any target. The nooperation representation is preferably implemented as a vector in the same vector space as thecontext representation. The value of the no-operation representation is preferably learned, i.e. consists of numbers stored as weights in the engagement means head 408, which are determined during training, preferably starting from a random selection.

[0088] In order to compare the context representation 406 to a processed target and / or nooperation representation 412, 414, 416, a matrix formed by the no-operation representation 416 and the processed target representations 412, 414 is preferably multiplied to a context representation 406 by matrix multiplication 418. The resulting vector consists in this case of scalar values 420, 422, 424, each of which is indicative of a similarity between the context representation 406 and one of the processed target representations 412, 414, or to the nooperation representation 416. The scalar value 420, 422, 424, corresponding to processed target 412, 414, or no operation 416, indicates to which extent the respective option is most conducive to a successful engagement strategy. Put differently, a scalar product is determined, which is one of the simplest similarity metrics available, in particular for representations in Euclidean spaces.

[0089] The target is then selected, 426, based on the scores. The selection of a target 426 may comprise a stochastic selection 426. Stochastic selection 426 preferably comprises selecting a subset of the targets related to the highest scores 420, 422, and performing a random or pseudo-random selection among the targets in the subset. During training, this has the advantage that variety of the output data is increased, which leads to faster convergence. The resulting output, the target selection 430, is an indication of a target to be engaged by the engagement means corresponding to the engagement means head 406.

[0090] The target selection 430 may have the format of an identifier of the target and is generally human-readable. It can be used as a control parameter to control the engagement system, or displayed, via an output device, to a user. For an engagement system as shown in Fig. 3, there is generally a plurality of engagement means heads 406, each of which corresponds to one engagement means, so that a target selection can be determined differently for each head, and the system may apply strategies in which each engagement means attacks a distinct target or a plurality of engagement means attack a same target.

[0091] Fig. 5 is a schematic drawing of a scenario. The scenario comprises a plurality of targets 502. In the context of air defence, the targets may comprise aerial vehicles, e.g. unmanned aerial vehicles (UAV), manned aerial vehicles, or missiles. The method is particularly advantageous in case the targets are fast moving targets, such as UAVs, and if there is a large number of targets. The targets 502 in this example comprise low value targets504 and a high value target 506. The engagement system 510 is an air defence system in this example. The engagement system 510 is generally operable to destroy one or more of the targets 502 or to otherwise nullity or reduce the effectiveness of the attack by the targets 502. In this example, the engagement system is operable to engage the targets as long as they are airborne. In other examples, the engagement system may be operable to engage land, sea, or underwater targets.

[0092] The engagement system 510 comprises sensors 516, which is a radar device in this example. In different embodiments, the sensor 516 may comprise a camera, sonar, or lidar system, or other sensor. However, also other sensor devices are possible, including passive sensors such as cameras, or other active sensors, such as sonar or lidar devices. The sensor 516 is configured to detect at least a part of the targets 502. Although Fig. 4 shows only targets that are detected, there may be further undetected targets (not shown). For the purpose of a partial observation reinforcement learning problem, however, only the targets that are detected by the sensors 516 are part of the observation 300.

[0093] The engagement system 510 further comprises engagement means 512, 514. An engagement means may generally include a weapon. The engagement means 512, 514 in this example comprise anti-aircraft artillery 512 and missile launchers 514. In other examples, engagement system 510 may comprise other engagement means such as radiation weapons, jamming systems, or other electronic attack devices. The engagement means 512, 514 are controlled by the agent component 522 and operable to engage targets 502 and thereby act on the environment. The engagement means may be ground-based, sea-based, air-based, spacebased, or underwater-based. The engagement means 512, 514 are preferably configured to execute any action command by the agent component. More specifically, each engagement means is preferably communicatively coupled to a delay head of the agent component and to execute any action commands from the corresponding delay head. Each engagement means is preferably configured not to engage any target if it does not receive any action command. This allows implementing a strategy where no targets are engaged during a time interval without the need of sending no-operation commands to the engagement means regularly.

[0094] Both sensor 516 and engagement means 512, 514, are part of the environment 500 in that they can be rendered inoperable, i.e. damaged or destroyed, by other influences, in particular as a result of being engaged by targets 502. This is indicated by alive status 304 for the engagement means, and alive status 308 of the sensor.

[0095] The sensor 516 and engagement means 512, 514, are also parts of the engagement system 510, which further comprises a command centre 518. The command centre 518 is the central part of the engagement system 510 and configured to control the sensor 516 and engagement means 512, 514. It comprises an agent component 522 operable to execute method 100 during operation. It comprises a preprocessor 520 configured to preprocess the signals detected from the sensor 516, which are radio frequency signals in this case, and to output a sensor alive status, which may be a Boolean value set to True if the sensor 516 continues to send information, and target states as described with respect to Fig. 3. The output is sent to agent component 522, which receives it as part of the observations. The agent component 522 is in bidirectional communication with the engagement means 512, 514. The agent component 522 thereby obtains information on the status of each of the engagement means 512, 514, in particular on whether it is still operating and the amount of ammunition available. The agent component 522 can, by executing method 100, determine an action and send the action commands to the engagement means 512, 514. The agent component 522 preferably comprises a machine learning algorithm, more preferably the neural network shown in Fig. 3. The environment may further comprise a protected asset 526 which is, in this embodiment, not part of the engagement system. A successful engagement of the protected asset by the targets may, however, result in negative terminal rewards by reward function 524. This allows training the agent component to develop a policy that protects asset 526, which may be a high-value asset, e.g. civilian or military infrastructure.

[0096] The agent component 522 may also be trained while deployed in the engagement system 510. To do so, it is necessary to apply a reward function 524, which also takes the sensor data as an input and determines a success, after each step in time and at the end of the training, if the action was successful and inputs a reward into the agent component 522. The reward function may be an algorithm that contains general functions to determine the rewards. It is, however, preferred to use simulations for training. In this case, the environment 500 is part of a simulation, and the target states are preferably entered into the agent component 522 directly, thus foregoing the step of detection by the sensor 516 and preprocessing. However, the sensor 516 is part of the simulated environment, and loss of the sensor results in a negative reward determined by the reward function 524. Once trained, the agent component 522 can be deployed into a real-world system. The protected asset 526 may be indicated to the agent component.Examples

[0097] Example 1. A computer-implemented method (100) of selecting a target (502) for an engagement system (510), the method comprising: determining at least one engagement means (512, 514) comprised in the engagement system (510), wherein the engagement means (512, 514) is configurable to engage one or more targets (502); determining a plurality of target representations ( 12, 414), wherein each target representation (412, 414) is indicative of a position, and preferably movement, of one of the targets (502) relative to the engagement means (512, 514); determining, by an agent component (320, 522) comprised in the engagement system (510), a context representation (406) indicative of a current and / or past situational context based on target information on one or more of the targets (502), engagement means information on one or more of the engagement means (512, 514), and / or protected assets information on one or more protected assets (526), preferably based on indications of positions of the one or more targets (502), a status of one or more of the engagement means (512, 514), and more preferably also one or more protected assets (526); comparing, by the agent component (320, 522), each of the target representations (412, 414) to the context representation (406) to determine a respective score (422, 424) for each of the targets (502), in particular wherein the score (422, 424) is indicative of a similarity of the respective target representation to the context representation (406); determining, based on the scores (422, 424), a target selection (430) for the engagement means (512, 514); and transmitting the target selection (430) to the engagement means (512, 514).

[0098] Example 2. The method of example 1, wherein the engagement means (512, 514) is configurable to engage at most one target of the one or more targets (502) at a time; wherein higher scores are indicative of a higher priority of engaging the target; wherein higher scores are indicative of a higher similarity of the corresponding target representation to the context representation (406); and / or wherein determining the target selection (430) comprises selecting a target related to a maximum score.

[0099] Example 3. The method of any of the preceding examples, wherein the target representations (412, 414) comprise a no-operation representation (416) indicative of an instruction not to engage any target.

[0100] Example 4. The method of any of the preceding examples, wherein the context representation (406) is determined by a recurrent neural network, RNN (356), preferably a long short term memory, LSTM, comprised in the agent component (320, 522) based on a situational awareness representation (352) based on positions, and preferably movement, of targets (502) and / or engagement means (512, 514).

[0101] Example 5. The method of any of the preceding examples, wherein the or a situational awareness representation (352) is determined by processing one or more of the target representations (338, 340) and, preferably, one or more engagement means representations (342, 344).

[0102] Example 6. The method of any of the preceding examples, wherein the or a situational awareness representation (352) comprises a mean embedding (348) of the target representations (338, 340) and one or more engagement means representations (342, 344) indicative of a status of each of the engagement means (512, 514).

[0103] Example 7. The method of any of the preceding examples, wherein the or a situational awareness representation (352) comprises, and preferably is, a concatenation of the mean embedding (348) and an asset representation of properties of the one or more protected assets.

[0104] Example 8. The method of any of the preceding examples, wherein the target representations (338, 340) and / or the one or more engagement means representations (342, 344) are determined by a backbone machine learning algorithm (334) based on input representations, the backbone machine learning algorithm preferably comprising a self-attention based backbone neural network, more preferably an encoder of a transformer-type neural network, and wherein the input representations are preferably based on, more preferably comprise, sensor data indicative of: positions, and preferably movement, of the one or more targets (502); and / or a status of the engagement means (512, 514).

[0105] Example 9. A computer-implemented method (200) of training an agent component (320, 522), in particular an agent component (320, 522) comprising a machine learning algorithm of any of the preceding examples, wherein the agent component (320, 522) pertains to the or an engagement system (510) comprising at least one engagement means (512, 514) configurable to engage one or more targets (502), and wherein the method comprises: determining training data comprising observations of an environment indicative of a position, and preferably movement, ofthe one or more targets (502) relative to the one or more engagement means (512, 514); determining, by the agent component (320, 522) and based on the training data, a target selection (430) for the engagement means (512, 514); and / or updating the agent component (320, 522) based on the reward.

[0106] Example 10. The method of any of the preceding examples, wherein determining a target selection comprises stochastically selecting one or more of the one or more targets (502).

[0107] Example 11. The method of any of the preceding examples, wherein the selfattention based machine learning algorithm forms part of a reinforcement learning based agent component (320, 522), and / or wherein the agent component (320, 522) preferably comprises: the or a backbone machine learning algorithm configured to determine the target representations (338, 340) and engagement means representations (342, 344); the or a recurrent neural network configured to determine the context representation based on the target representations (338, 340) and engagement means representations (342, 344); and / or at least one, preferably a plurality of, the target selection algorithms as output heads, wherein each of the target selection heads is configured to determine an action command for one of the engagement means (512, 514).

[0108] Example 12. The method of any of examples 1-8, wherein the agent component has been trained according to the method of any of examples 9-11.

[0109] Example 13. A system comprising one or more processors and one or more storage devices, wherein the system is configured to perform the computer- implemented method of any one of examples 1-12.

[0110] Example 14. A computer program product for loading into a memory of a computer, comprising instructions, that, when executed by a processor of the computer, cause the computer to execute a computer-implemented method of any of examples 1-12.

[0111] Example 15. A computer program product comprising a trained machine learning module obtainable by the computer-implemented method of any of examples 9-11.Reference signs100 Method of controlling an engagement system102-120 Steps of method 100200 Method of generating an agent component of an engagement system 202-216 Steps of method 200300 Observation302 Target status304, 306 Target input representations308 Engagement means status310, 312 Engagement means representations314 Protected assets status316, 318 Protected assets representations320 Agent component322, 324, 326, 328, 330, 332 Multilayer perceptrons334 Backbone layer336 Target representation dataset338, 340 Target representations342, 344 Engagement means representations346 Mean calculation348 Mean embedding350 Concatenation352 Situational awareness representation354 Multilayer perceptron356 Recurrent neural network358 Output layer360, 362 Engagement means head364 Target selection366, 368 Target selections for respective engagement means400 Target representation dataset402, 404 Target representations406 Context representation408 Engagement means headMLP, 414 Target Representations No-operation representation Matrix multiplication No-operation score, 424 Target scoresSelection operation StochasticsTarget election EnvironmentTargetsLow-value targetHigh-value target Engagements Engagement system, 514 Engagement means SensorCommand centre PreprocessorAgent component Reward function Protected asset

Claims

29Claims1. A computer-implemented method (100) of selecting a target (502) for an engagement system (510), the method comprising:determining at least one engagement means (512, 514) comprised in the engagement system (510), wherein the engagement means (512, 514) is configurable to engage one or more targets (502);determining a plurality of target representations (412, 414), wherein each target representation (412, 414) is indicative of a position, and preferably movement, of one of the targets (502) relative to the engagement means (512, 514);determining, by an agent component (320, 522) comprised in the engagement system (510), a context representation (406) indicative of a current and / or past situational context based on target information on one or more of the targets (502), engagement means information on one or more of the engagement means (512, 514), and / or protected assets information on one or more protected assets (526), preferably based on indications of positions of the one or more targets (502), a status of one or more of the engagement means (512, 514), and more preferably also one or more protected assets (526); comparing, by the agent component (320, 522), each of the target representations (412, 414) to the context representation (406) to determine a respective score (422, 424) for each of the targets (502), in particular wherein the score (422, 424) is indicative of a similarity of the respective target representation to the context representation (406); determining, based on the scores (422, 424), a target selection (430) for the engagement means (512, 514); andtransmitting the target selection (430) to the engagement means (512, 514).

2. The method of claim 1,wherein the engagement means (512, 514) is configurable to engage at most one target of the one or more targets (502) at a time;wherein higher scores are indicative of a higher priority of engaging the target;wherein higher scores are indicative of a higher similarity of the corresponding target representation to the context representation (406); and / orwherein determining the target selection (430) comprises selecting a target related to a maximum score.

303. The method of any of the preceding claims, wherein the target representations (412, 414) comprise a no-operation representation (416) indicative of an instruction not to engage any target.

4. The method of any of the preceding claims, wherein the context representation (406) is determined by a recurrent neural network, RNN (356), preferably a long short term memory, LSTM, comprised in the agent component (320, 522) based on a situational awareness representation (352) based on positions, and preferably movement, of targets (502) and / or engagement means (512, 514).

5. The method of any of the preceding claims, wherein the or a situational awareness representation (352) is determined by processing one or more of the target representations (338, 340) and, preferably, one or more engagement means representations (342, 344).

6. The method of any of the preceding claims, wherein the or a situational awareness representation (352) comprises a mean embedding (348) of the target representations (338, 340) and one or more engagement means representations (342, 344) indicative of a status of each of the engagement means (512, 514).

7. The method of any of the preceding claims, wherein the or a situational awareness representation (352) comprises, and preferably is, a concatenation of the mean embedding (348) and an asset representation of properties of the one or more protected assets.

8. The method of any of the preceding claims, wherein the target representations (338, 340) and / or the one or more engagement means representations (342, 344) are determined by a backbone machine learning algorithm (334) based on input representations, the backbone machine learning algorithm preferably comprising a self-attention based backbone neural network, more preferably an encoder of a transformer-type neural network, and wherein the input representations are preferably based on, more preferably comprise, sensor data indicative of:positions, and preferably movement, of the one or more targets (502); and / ora status of the engagement means (512, 514).

9. A computer-implemented method (200) of training an agent component (320, 522), in particular an agent component (320, 522) comprising a machine learning algorithm of any of the preceding claims, wherein the agent component (320, 522) pertains to the or an engagement system (510) comprising a plurality of engagement means (512, 514) configurable to engage one or more targets (502), and wherein the method comprises: determining training data comprising observations of an environment indicative of a position, and preferably movement, of the one or more targets (502) relative to the engagement means (512, 514);determining, by the agent component (320, 522) and based on the training data, a target selection (430) for each of the engagement means (512, 514); and / orupdating the agent component (320, 522) based on single-agent reinforcement learning.

10. The method of any of the preceding claims, wherein determining a target selection comprises stochastically selecting one or more of the one or more targets (502).

11. The method of any of the preceding claims, wherein the self-attention based machine learning algorithm forms part of a reinforcement learning based agent component (320, 522), and / or wherein the agent component (320, 522) preferably comprises:the or a backbone machine learning algorithm configured to determine the target representations (338, 340) and engagement means representations (342, 344);the or a recurrent neural network configured to determine the context representation based on the target representations (338, 340) and engagement means representations (342, 344); and / orat least one, preferably a plurality of, the target selection algorithms as output heads, wherein each of the target selection heads is configured to determine an action command for one of the engagement means (512, 514).

12. The method of any of claims 1-8, wherein the agent component has been trained according to the method of any of claims 9-11.

13. A system comprising one or more processors and one or more storage devices, wherein the system is configured to perform the computer-implemented method of any one of claims 1-12.

14. A computer program product for loading into a memory of a computer, comprising instructions, that, when executed by a processor of the computer, cause the computer to execute a computer-implemented method of any of claims 1-12.

15. A computer program product comprising a trained machine learning module obtainable by the computer-implemented method of any of claims 9-11.

Images