Distributed execution environment and method for executing a plurality of applications on a plurality of computing nodes in order to control a cyber-physical system

Abstract

The invention relates to a distributed execution environment (100) for executing a plurality of applications on a plurality of computing nodes in order to control a cyber-physical system (CPS), comprising: - a risk assessment unit (101) designed to receive risk assessment data relating to the cyber-physical system and determine, on the basis of the risk assessment data, a risk value and output same, - a storage unit (102) designed to provide a plurality of different provision specifications, each of which has at least one rule for executing at least some of the applications on selected computing nodes, and - a management unit (103) designed to select, on the basis of the risk value, a provision specification and apply same, wherein the plurality of applications are distributed to the corresponding computing nodes in accordance with the at least one rule of the selected provision specification in order to control the cyber-physical system.

Description

[0001] 202502009

[0002] 1

[0003] Description

[0004] Distributed execution environment and methods for running multiple applications on multiple compute nodes to control a cyber-physical system

[0005] The invention relates to a distributed execution environment and a computer-implemented method for executing a plurality of applications on a plurality of computing nodes for controlling a cyber-physical system, as well as a computer program product.

[0006] For industrial systems or cyber-physical systems (CPS), requirements for so-called denial-of-service (DoS) resilience are becoming increasingly relevant. In particular, essential functions of a CPS should be maintained even during a DoS attack and / or (limited) continued operation should be enabled, preferably while avoiding overloading other (sub)systems.

[0007] In a cyber-physical system (CPS), control functionality is implemented through multiple software-based workloads or applications, such as apps, containers, and virtual machines (VMs). These workloads can run on different compute nodes, for example, on app-enabled control units located locally within the technical process, on edge compute nodes, or in cloud-based backend systems.

[0008] Traditional distributed execution environments typically address data centers where the relevant functionality resides within the workloads themselves. In contrast, Cyber-Physical Systems (CPS) aim to ensure the availability of the CPS's technical processes. This usually requires the combined functionality of multiple workloads. Furthermore, CPS workloads must meet real-time requirements. Conventional workload schedulers typically address only overload situations or random outages, but not attack scenarios.

[0009] It is therefore an object of the present invention to make the implementation of workloads / applications of a cyber-physical system resilient and, in particular, to improve it in the event of a DoS attack.

[0010] 2

[0011] The problem is solved by the measures described in the independent claims. Advantageous embodiments of the invention are described in the dependent claims.

[0012] According to a first aspect, the invention relates to a distributed execution environment for running a plurality of applications on a plurality of computing nodes for controlling a cyber-physical system (CPS), comprising:

[0013] - a risk assessment unit that is set up to receive risk assessment data from the CPS and to determine and output a risk value based on the risk assessment data,

[0014] - a storage unit configured to provide a plurality of different deployment rules, each comprising at least one rule for running at least some of the applications on selected compute nodes, and

[0015] - an administrative unit that is set up to select and apply a provisioning rule depending on the risk value, whereby the majority of applications are distributed to the appropriate compute nodes according to at least one rule of the selected provisioning rule for controlling the CPS.

[0016] A CPS can be, for example, an industrial automation system, a machine, an autonomous transport system (AGV), a production system, or a robot.

[0017] In particular, a CPS can be controlled by a multitude of applications / workloads that can run on different computing nodes, e.g., on edge devices. A distributed execution environment is provided for this purpose.

[0018] A risk value can be, in particular, a numerical value, a discrete value, a vector, or a matrix.

[0019] A workload deployment template can, in particular, include at least one policy that determines which application is executed on which compute node. Specifically, a workload deployment template can also include a combination of an application selection rule and a scheduler rule.

[0020] It is an advantage of the present invention that the execution of applications for controlling a CPS depending on a risk assessment regarding network attacks202502009

[0021] 3

[0022] This can be regulated. In particular, this ensures the continued operation of the CPS and makes the CPS control system more resilient to DoS attacks.

[0023] The distributed execution environment distributes the respective applications / workloads across designated compute nodes according to a selected deployment policy, which is chosen based on a risk assessment. The respective deployment policies are preferably adapted to various attack scenarios.

[0024] By providing multiple deployment rules, different workload implementation variants can be used, instead of only distributing and executing predefined workloads across compute nodes. Furthermore, execution rules can be defined flexibly.

[0025] Furthermore, deployment protocols can be tested in advance, thus avoiding unforeseen effects such as overload situations. This also prevents an attacker from bringing the CPS into an undefined or untested state.

[0026] Furthermore, the allocation of multiple applications within a CPS is coordinated according to a deployment protocol and is therefore testable. Testing can be performed, for example, in a dedicated test environment, such as a CPS simulation environment (digital twin), or on the actual, operational CPS system (e.g., during commissioning, acceptance testing). Because the various deployment protocols are prepared and testable, a backup solution can be selected and activated independently in the event of an attack, even without specialized personnel with specific IT and security expertise, in order to maintain CPS operation.

[0027] In one embodiment of the invention, the administrative unit can be configured to select the provisioning instruction based on a predefined selection rule.

[0028] In one embodiment of the invention, the administrative unit can be configured to select the provisioning requirement using artificial intelligence.

[0029] In one embodiment of the invention, the distributed execution environment can further include a confirmation unit configured to receive confirmation information for the selected deployment instruction and to apply the selected deployment instruction after the confirmation.

[0030] 4

[0031] This ensures, for example, that a change to a different deployment policy and thus a modified execution of the applications according to this deployment policy can be checked again in advance, e.g. by a user.

[0032] In one embodiment of the invention, the administrative unit can be further configured to perform the selection and execution of a provisioning instruction on an event-based basis.

[0033] For example, the risk assessment can be carried out continuously, with a change to a different provisioning rule only occurring in predefined cases.

[0034] In one embodiment of the invention, the administrative unit can be further configured to perform the selection and execution of a provisioning instruction at predetermined times.

[0035] In one embodiment of the invention, the risk assessment data may include the following:

[0036] - Network monitoring data from a communication network used by the CPS,

[0037] - Data from a network-based attack detection system of the communication network, - Monitoring data from monitoring the existing computing nodes,

[0038] - Monitoring data from monitoring the distributed execution environment,

[0039] - Data from vulnerability databases,

[0040] and / or

[0041] - Security situation information.

[0042] In one embodiment of the invention, a provision requirement may include the following:

[0043] - a rule for selecting at least one application from the majority of applications, and / or

[0044] - a rule for distributing an application across a computing node.

[0045] In one embodiment of the invention, a provisioning instruction can merely select applications and assign them to predetermined computing nodes that are necessary for emergency operation of the CPS.

[0046] 5

[0047] For example, in this way an execution of the respective applications of the CPS can be defined for emergency operation, whereby the use of the corresponding provisioning rule is preferably event-based.

[0048] In one embodiment of the invention, a provision requirement can be directed towards different types of applications.

[0049] In one embodiment of the invention, the storage unit can be configured to additionally provide a plurality of further deployment instructions, each comprising at least one rule for executing at least a part of the applications on different execution environments of different types, and furthermore, the management unit can be configured to select and apply one of these further deployment instructions depending on the risk value, wherein, according to the at least one rule of the selected further deployment instruction, the part of the applications is executed on the corresponding execution environment.

[0050] For example, it is possible to switch between different workload technologies, such as from containers to virtual machines (VMs). Different types of runtime environments (RTEs) can be provided for this purpose, for example, for running containers or for running VMs. In this scenario, it is also possible that, in addition to an application deployment policy, there is also an associated deployment policy for the runtime environment, based on which the required runtime environments / RTEs are set up or activated on the compute nodes.

[0051] According to a second aspect, the invention relates to a computer-implemented method for executing a plurality of applications on a plurality of computing nodes for controlling a cyber-physical system (CPS) comprising the following method steps:

[0052] - Receiving risk assessment data from the CPS,

[0053] - Determining and outputting a risk score based on the risk assessment data,

[0054] - Providing a plurality of different deployment rules, each including at least one rule for running at least some of the applications on selected compute nodes,

[0055] - Selecting a provisioning rule depending on the risk value

[0056] and202502009

[0057] 6

[0058] - Applying the selected deployment rule, whereby the majority of applications are distributed to the appropriate compute nodes according to at least one rule of the selected deployment rule to control the CPS.

[0059] Furthermore, the invention relates to a computer program product that can be directly loaded into a programmable computer, comprising program code parts which, when the program is executed by a computer, cause it to perform the steps of a method according to the invention.

[0060] A computer program product can be provided or delivered from a server in a network, for example, on a storage medium such as a memory card, USB stick, CD-ROM, DVD, a non-volatile / permanent storage medium, or in the form of a downloadable file.

[0061] Exemplary embodiments of the invention are shown in the drawings and are explained in more detail below. The drawings show:

[0062] Fig. 1: a first embodiment of a distributed implementation environment according to the invention;

[0063] Fig. 2: a second first embodiment of a distributed implementation environment according to the invention; and

[0064] Fig. 3: an embodiment of a computer-implemented method according to the invention for executing a plurality of applications on a plurality of computing nodes for controlling a cyber-physical system.

[0065] Corresponding parts are marked with the same reference symbols in all figures.

[0066] In particular, the following embodiments merely show exemplary implementation possibilities of how such implementations of the teaching according to the invention could look, since it is impossible and also not helpful or necessary for understanding the invention to name all these implementation possibilities.

[0067] Furthermore, a (relevant) person skilled in the art, with knowledge of the method claim(s), is aware of all possibilities customary in the prior art.

[0068] 7

[0069] The means for realizing the invention are of course known, so that in particular a separate disclosure in the description is not required.

[0070] Figure 1 shows a first embodiment of a distributed execution environment 100 for running a plurality of applications WL on a plurality of compute nodes CN for controlling a cyber-physical system (CPS). A CPS can be, for example, an industrial automation system.

[0071] The execution environment 100 comprises a multitude of compute nodes CN, each with its own runtime environment (RTE). Various applications or workloads WL, such as applications / apps, containers, and / or virtual machines (VMs), can be adaptively executed on these compute nodes CN. The distributed execution environment 100 includes, in particular, a scheduler 300, which is configured to determine which application WL can be executed on which compute node CN in order to control the CPS.

[0072] The distributed execution environment 100 further comprises a risk assessment unit 101, a storage unit 102, and the management unit 103. Additionally, the distributed execution environment 100 may include an acknowledgment unit 104.

[0073] Additionally, an API manager may be provided that forwards requests from clients (CL) based on a current mapping of applications (WL) of the CPS to compute nodes for processing and returns the corresponding responses. However, there may also be no API manager, in which case the applications perform calculations independently.

[0074] The units of the distributed execution environment 100 and the compute nodes CN are interconnected via a network / communication network NW.

[0075] The risk assessment unit 101 is configured to receive risk assessment data RD from the cyber-physical system and to determine and output a risk value RV based on the risk assessment data RD.

[0076] The risk assessment data (RD) can be compiled from various sources, such as different risk information providers 201, 202. The risk assessment data (RD) can include at least one of the following pieces of information: 202502009

[0077] 8

[0078] Network monitoring data of a communication network NW used by the cyber-physical system, such as network monitoring, for QoS monitoring of the communication network NW used, in particular to detect DoS attacks;

[0079] Data from a network-based intrusion detection system (NIDS) of the communication network NW, whereby data transmission can be monitored network-based and analyzed for potentially malicious network traffic;

[0080] Monitoring data from monitoring the available computing nodes CN, e.g., so-called host monitoring for performance monitoring of the used computing nodes CN, whereby, for example, ongoing DoS attacks against individual computing nodes CN can be detected (e.g., by monitoring CPU or memory utilization);

[0081] Monitoring data from monitoring the distributed execution environment 100, such as a Security Operation Center that monitors the operation of the distributed execution environment 100;

[0082] Data from vulnerability databases, such as https: / / www.cve.org / , https: / / nvd.nist.gov / vuln or https: / / osv.dev / ;

[0083] and / or

[0084] Security situation information, such as from a Security Data Lake with AI-supported data analysis.

[0085] Storage unit 102 is configured to provide a plurality of different deployment rules DT, each of which includes at least one rule for running at least some of the applications WL on selected compute nodes CN.

[0086] A provisioning instruction DT preferably includes the following:

[0087] - a rule for selecting at least one application WL from the majority of applications,

[0088] and / or

[0089] - a rule for distributing at least one WL application to a CN compute node.

[0090] A deployment rule DT, for example, specifies which predefined workloads WL are executed on which individual compute node CN or group of compute nodes. Distribution is then handled by Scheduler 300. A deployment rule DT can also be defined as the combination of a specific workload.

[0091] 9

[0092] Selection policy and a specific scheduler policy are understood as follows. A deployment policy DT can statically specify a fixed, predefined assignment of workloads to compute nodes or grant the scheduler 300 a certain degree of flexibility, i.e., for a dynamic assignment to compute nodes, but only within the limits defined by the selected deployment policy DT*, which determines which workloads WL are selected for execution and on which compute nodes they may be executed.

[0093] A deployment specification (DT) comprises, in particular, a plurality of workload images and their allocation to compute nodes. A deployment specification (DT) can be directed toward different types of applications, such as the use of containers or virtual machines (VMs). The workloads / applications (WL) of a deployment specification (DT) can therefore be of the same type (e.g., VM, container, WebAssembly), but it is also possible for a single deployment specification (DT) to contain different workload types.

[0094] It is also possible that the provisioning rule DT is used to select applications and assign them to predetermined computing nodes that are necessary for emergency operation of the CPS.

[0095] Administrative Unit 103 is configured to select a deployment rule DT* from the set of available deployment rules DT, depending on the risk value RV, and then apply it. The majority or only a portion of the CPS's applications WL are then distributed to the corresponding compute nodes CN according to at least one rule of the selected deployment rule for controlling the cyber-physical system. Administrative Unit 103 is specifically configured to select the deployment rule DT* based on a predefined selection rule. This selection rule is primarily based on the risk assessment. Additionally or alternatively, Administrative Unit 103 can be configured to select the deployment rule using a suitably trained artificial intelligence.The selection and execution of a provisioning instruction by administrative unit 103 is carried out, for example, on an event-based basis or at predetermined times.

[0096] The selected deployment rule DT* is provided to Scheduler 300. The scheduler then distributes the corresponding applications WL of the industrial automation system CPS to the network according to the selected deployment rule DT*.

[0097] 10

[0098] prescribed compute node CN. In particular, by activating a selected provisioning rule DT*, a previously used / activated provisioning rule can be deactivated, whereby the corresponding applications can be terminated or deactivated according to the previously used / original provisioning rule.

[0099] A risk-based management unit 103 is therefore provided for managing the deployment rules DT, which determines the current risk situation based on the risk information of at least one risk information provider 201, 202. Depending on this, one of several deployment rules DT* is selected and set up by management unit 103 as the active deployment rule DT* for the scheduler 300.

[0100] The confirmation unit 104 is configured to receive confirmation information for the selected deployment rule DT* and to apply the selected deployment rule DT* after confirmation. For example, a deployment rule DT* can only be applied after manual confirmation by a user.

[0101] The selection of a deployment specification DT* can therefore be fully automated, e.g., based on a predefined selection rule or an AI-based selection component. However, it is also possible for the selection process to identify multiple deployment specifications and offer them to operational technology (OT) personnel for manual selection. This allows the prepared deployment specifications DT to be evaluated based on the current security situation to determine which deployment variant offers the best cybersecurity performance (CPS) for the currently known vulnerabilities and / or observed attack attempts at a defined, acceptable risk level.

[0102] The risk-based selection of a deployment policy (DT) can be used in various ways to improve the cyber resilience of a CPS:

[0103] Selection of different workload images for at least one workload: Using different deployment rules (DT), multiple image variants can be prepared for equivalent, similar functionality. Different workload images reference different workload images for equivalent or only essential functionality.

[0104] 11

[0105] Workload Technology Switching: It is still possible to switch between different workload technologies, e.g., from containers to VMs or to WebAssembly. Different types of runtime environments (RTEs) can be provided, e.g., for running containers or VMs. In this scenario, in addition to the deployment guidelines for various applications, there may be an associated RTE deployment template / guideline for RTEs, based on which the required RTEs can be set up on the compute nodes.

[0106] Different allocation to compute nodes (such as control unit, edge, backend): WL applications can be executed on different compute nodes (CN). For example, the distribution depends on predefined real-time properties, whereby certain functions can only be implemented locally near the field. Nevertheless, the operation of the CPS can be maintained, possibly with limitations, if, for example, only slower, less performant operation is possible when control is no longer performed locally but by a more remote instantiated workload. It is also possible to deactivate functions such as a user interface, e.g., a graphical user interface (HMI) for machine operation.

[0107] Figure 2 shows another embodiment of a distributed execution environment 100 for running a plurality of applications WL on a plurality of compute nodes CN, ECN, CLN, RIO for controlling a cyber-physical system. An industrial automation system is shown as an application example, which interacts with the real, physical world RW via sensors S and actuators A. For the control and / or monitoring functions of the CPS, in addition to a physical programmable logic controller (PLC; not shown), remote I / O modules RIO are also provided, each of which is controlled, for example, by a virtualized PLC. A virtualized PLC can, for example, be in the form of a virtual machine running on an edge computing platform.

[0108] Monitoring and control data is transmitted, for example, via a wireless control network (WLN), such as a 5G SNPN network (SNPN: standalone non-public network) or an industrial WLAN infrastructure. However, wired data transmission, e.g., via Ethernet, is also possible.

[0109] 12

[0110] Several different RTE workload execution environments are planned, e.g., on an edge compute node (ECN) or a cloud backend (CLN). The latter is connected via the internet (WEB), for example.

[0111] For example, a risk assessment is performed based on an internet-based risk information provider 201 and a risk information provider of the OT control network 202. The internet-based risk information provider 201 can, for example, provide information on current vulnerabilities, attacks observed on the internet, and / or attacks on the cloud infrastructure. The risk information provider 202 of the OT control network can, for example, determine risk information based on observation of network communication in the control network WLN, such as by evaluating current production planning.

[0112] As described in Fig. 1, the management unit 103 can determine the current risk situation based on the risk information from the risk information providers 201 and 202. Depending on this, the management unit selects one of several deployment rules DT* and configures it on the scheduler 300 as the active deployment rule DT* in order to control the CPS.

[0113] Figure 3 shows an embodiment of a computer-implemented method according to the invention for executing a plurality of applications on a plurality of computing nodes for controlling a cyber-physical system.

[0114] The procedure comprises the following steps:

[0115] In a first step, S1 receives risk assessment data from the cyber-physical system.

[0116] In the next step S2, a risk value is determined and provided based on the risk assessment data.

[0117] In the next step, S3 provides a plurality of different deployment rules, each including at least one rule for running at least some of the applications on selected compute nodes. 202502009

[0118] 13

[0119] In the next step S4, a provisioning rule is selected depending on the risk value.

[0120] In the next step S5, the selected provisioning rule is applied, whereby the majority of applications are distributed to the corresponding computing nodes according to at least one rule of the selected provisioning rule for controlling the cyber-physical system.

[0121] All described and / or illustrated features can be advantageously combined within the scope of the invention. The invention is not limited to the described embodiments.

Claims

202502009 14 Patent claims 1. Distributed execution environment (100) for running a plurality of applications (WL) on a plurality of compute nodes (CN) for controlling a cyber-physical system (CPS), comprising: a risk assessment unit (101) configured to receive risk assessment data (RD) of the cyber-physical system and to determine and output a risk value (RV) based on the risk assessment data, a storage unit (102) configured to provide a plurality of different deployment rules (DT), each comprising at least one rule for executing at least one part of the applications on selected compute nodes (CN), and an administrative unit (103) that is set up to select and apply a provisioning rule (DT*) depending on the risk value (RV), wherein the majority of applications are distributed to the appropriate compute nodes according to the at least one rule of the selected provisioning rule for controlling the cyber-physical system.

2. Distributed execution environment according to claim 1, wherein the management unit (103) is configured to select the deployment instruction based on a predefined selection rule.

3. Distributed execution environment according to one of the preceding claims, wherein the management unit (103) is configured to select the deployment instruction using artificial intelligence.

4. Distributed execution environment according to one of the preceding claims, further comprising a confirmation unit (104) configured to receive confirmation information for the selected deployment instruction and to apply the selected deployment instruction after the confirmation.

5. Distributed execution environment according to one of the preceding claims, wherein the management unit (103) is further configured to perform the selection and execution of a deployment instruction on an event-based basis. 15 6. Distributed execution environment according to one of the preceding claims, wherein the management unit (103) is further configured to perform the selection and execution of a deployment instruction at predetermined times.

7. Distributed execution environment according to any of the preceding claims, wherein the risk assessment data (RD) comprises: Network monitoring data from a communication network used by the cyber-physical system, Data from a network-based attack detection system of the communication network, Monitoring data from monitoring the existing computing nodes, monitoring data from monitoring the distributed execution environment, data from vulnerability databases, and / or Security situation information.

8. Distributed execution environment according to any of the preceding claims, wherein a deployment instruction (DT) comprises: a rule for selecting at least one application from the majority of applications, and / or a rule for distributing an application across a computing node.

9. Distributed execution environment according to one of the preceding claims, wherein a deployment specification (DT) merely selects and assigns applications to predetermined compute nodes that are necessary for emergency operation of the CPS.

10. Distributed execution environment according to one of the preceding claims, wherein a deployment instruction (DT) is directed to different types of applications (WL).

11. Distributed execution environment according to one of the preceding claims, wherein the storage unit is configured to additionally provide a plurality of further deployment rules, each comprising at least one rule for executing at least a part of the applications on different execution environments of different types. and202502009 16 wherein the administrative unit is set up to select and apply one of these further deployment rules depending on the risk value, wherein, in accordance with at least one rule of the selected further deployment rule, part of the applications are executed on the corresponding execution environment.

12. Computer-implemented method for executing a plurality of applications on a plurality of computing nodes for controlling a cyber-physical system, comprising the following process steps: Receiving (S1) risk assessment data from the cyber-physical system, determining (S2) and outputting a risk score based on the risk assessment data, providing (S3) a plurality of different provisioning rules, each comprising at least one rule for executing at least some of the applications on selected computing nodes, Selecting (S4) a provisioning rule depending on the risk value and Applying (S5) the selected provisioning rule, whereby the majority of applications are distributed to the appropriate compute nodes according to the at least one rule of the selected provisioning rule for controlling the cyber-physical system.

13. Computer program product that can be directly loaded into a programmable computer, comprising program code segments suitable for performing the steps of the method according to claim 12.

Images