Methods and systems for conflict management

Abstract

A conflict management method is provided. The method comprises identifying a conflict based on: (i) at least one control action instigated by an intent of an application, and / or (ii) one or more environment observables of the network; determining whether a nature of the conflict is direct, indirect or implicit based on the control action and / or the environment observables; identifying one or more causes of the conflict based on the nature of the conflict, the control action and / or the environment observables; prioritising the one or more causes of the conflict based on a predetermined prioritisation framework; and sending guidance to the application based on the conflict, the one or more causes of the conflict, and / or the priority of the one or more causes of the conflict, wherein the guidance uses a perturbative harmonization approach to guide changes into the network to reduce severity of the conflict.

Description

[0001] METHODS AND SYSTEMS FOR CONFLICT MANAGEMENT

[0002] FIELD OF THE INVENTION

[0003] Embodiments of the present invention described herein relate to methods and systems for conflict management within a network infrastructure, and in particular to a computer-implemented method, a computer program, and an information programming apparatus.

[0004] BACKGROUND OF THE INVENTION

[0005] As network infrastructure evolves and offers app hosting capabilities, certain complexities arise when multiple apps attempt to change the same network segment or service segment concurrently for different intent. This potentially leads to several combinations of direct, indirect and implicit conflict situations.

[0006] It is desirable to be able to improve conflict management capabilities within a hosted network infrastructure.

[0007] SUMMARY OF THE INVENTION

[0008] It is an aim of the present disclosure to at least partially address one or more of the challenges mentioned above. The invention is defined in the independent claims, to which reference should now be made. Further features are set out in the dependent claims.

[0009] According to an embodiment of a first aspect there is disclosed herein a conflict management method for managing conflicts within a network infrastructure, the method comprising: identifying a conflict based on: (i) at least one control action instigated by an intent of an application, and / or (ii) one or more environment observables of the network; determining whether a nature of the conflict is direct, indirect or implicit based on the control action and / or the environment observables; identifying one or more causes of the conflict based on the nature of the conflict, the control action and / or the environment observables; prioritising the one or more causes of the conflict based on a predetermined prioritisation framework; and sending guidance to the application based on the conflict, the one or more causes of the conflict, and / or the priority of the one or more causes of the conflict, wherein the guidance uses a perturbative harmonization approach to guide changes into the network to reduce severity of the conflict. In some embodiments, the method steps are performed by a multi-agent reinforcement learning, MARL, model comprising: a plurality of agents, wherein the intent of the application is associated with an agent of the plurality of agents, and wherein each of the plurality of agents is associated with a corresponding intent; and a central module which communicates with each of the plurality of agents and monitors an environment of the network to obtain the environment observables.

[0010] In some embodiments, the agent comprises a learning agent configured to implement a Markov process for reinforcement learning for its corresponding intent.

[0011] In some embodiments, the agent comprises a guidance agent configured to obtain central guidance from the central module and to obtain local guidance by communicating with the learning agent, wherein the guidance comprises the local guidance and the central guidance.

[0012] In some embodiments, the central module comprises a deep neural network configured to implement a Markov process for reinforcement learning.

[0013] In some embodiments, the steps of determining the nature of the conflict and identifying one or more causes of the conflict are performed using the deep neural network which is trained to recognise relationships between variables in an environment of the network infrastructure within a purview of the intent.

[0014] In some embodiments, the relationships between variables comprise one or more of a relationship between: (i) a plurality of environment observables and a plurality of control actions, wherein the plurality of environment observables comprise the one or more environment observables, and wherein the plurality of control actions comprise the at least one control action; (ii) the plurality of control actions and a plurality of rewards, wherein the plurality of rewards relate to the Markov process; (iii) the plurality of control actions and a plurality of rewards with penalties, wherein the plurality of rewards with penalties relate to the Markov process.

[0015] In some embodiments, the relationships between variables comprise mappings of one variable against another. In some embodiments, the guidance comprises one or more of: (i) changing a frequency of monitoring of the environment observables; (ii) changing the predefined time interval of the perturbative harmonization approach; (iii) changing the direction of the control action; (iv) pausing the control action; (v) delaying the control action; and (vi) withdrawing the control action.

[0016] In some embodiments, the method further comprises prompting a user to intervene if the conflict cannot be resolved.

[0017] In some embodiments, the predetermined prioritisation framework is based on CAP theorem, and preferably an adapted CAP theorem for managing trade-off for app hosting network infrastructure.

[0018] In some embodiments, the perturbative harmonization approach comprises: (i) actioning a first change; (ii) waiting for a predefined time interval; (iii) evaluating the effect of the first change; (iv) determining a direction of a second change, based on the effect of the first change; and (v) actioning a second change.

[0019] According to an embodiment of a second aspect there is disclosed herein a computer program which, when run on a computer, causes the computer to carry out a method according to any of the embodiments described above.

[0020] According to an embodiment of a third aspect there is disclosed herein an information processing apparatus comprising a memory and a processor connected to the memory, wherein the processor is configured to perform a method according to any of the embodiments described above.

[0021] Features relating to any aspect / embodiment may be applied to any other aspect / embodiment.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments of the invention will now be further described by way of example only and with reference to the accompanying drawings, wherein like reference numerals refer to like parts, and wherein: Figure 1 is a flowchart illustrating an embodiment of the invention;

[0024] Figure 2 is a diagram illustrating CAP theorem and adapted CAP theorem;

[0025] Figure 3 is a diagram illustrating a multi-agent reinforcement learning architecture;

[0026] Figure 4 is a diagram illustrating how an individual agent interacts with the environment;

[0027] Figure 5 is a flowchart illustrating app / intent onboarding and learning and guidance execution flow;

[0028] Figure 6 is a flowchart illustrating the critic DNN’s conflict management approach;

[0029] Figure 7 is a diagram illustrating how the critic DNN, the agent monitor, the environment monitor and the replay buffer and database interact with one another within the central critic framework; and

[0030] Figure 8 is a diagram illustrating an apparatus.

[0031] DETAILED DESCRIPTION

[0032] This invention provides conflict management capabilities within a hosted network infrastructure, while offering freedom of implementation to external app vendors or intent owners. The invention may involve: (i) a trade-off managed platform / environment for app-hosting network infra, (ii) a perturbative harmonization approach to allow introducing changes into the platform and infrastructure from the apps and intent owners, and (iii) an adapted multi-agent reinforcement learning capabilities leveraging trade-off managed environment and perturbative harmonization for detecting and resolving direct, indirect and implicit conflicts. With this approach, the apps and intent owners remain autonomous while working over app-hosting network infrastructure and platform but are provided with a guidance framework that can steer them towards a non-conflicting situation, as and when needed.

[0033] This invention may be used in any sector which may benefit from conflict management capabilities within a hosted network infrastructure. Healthcare apps, financial apps or autonomous car apps are a few examples of applications which may benefit from embodiments of the invention. In general, any domain specific application that leverages the underlying network infrastructure may benefit from embodiments of the invention.

[0034] Advantages of the invention include:

[0035] • Seamless detection and resolution of the conflicts introduced by concurrent functioning of apps, thus improving the operation stability of the network infrastructure, reducing the operation and maintenance cost of the network and promoting the seamless deployment and operations of the intelligent applications. • The trade-off managed environment establishes appropriate enforcement of priorities for conflicting control actions, and appropriate correlation of cause and effect for apparently unrelated control actions, improving the stability and predictability of the network behaviour.

[0036] • The perturbative harmonization approach ensures that control action exerted on the network infrastructure by an app does not bring a sudden disturbance on the overall infrastructure and service, and by dividing the expected magnitude of change into smaller degrees of periodic actions, further unmanaged impacts are not introduced, improving operational efficiency and cutting down on cascading effects of operational overhead.

[0037] Various aspects and details of these principal concepts will be described below with reference to Figures 1 to 8.

[0038] Applications operating within a Hosting Network Infrastructure that adheres to a particular standard often share functional areas. Similarly, applications deployed across different standards-aligned infrastructures may also operate within overlapping functional areas on the same network infrastructure. When multiple applications function within the same network segment and functional purview, the potential for operational overlap, and thus conflicts, increases.

[0039] Functional overlaps across individual apps may trigger conflicts. For example, direct conflicts may include: • Two or more apps requesting different network settings for the very same network or service segment and configuration profiles.

[0040] • A new request from an app may conflict with the running configuration resulting from a previous request of another app.

[0041] • Concurrent resource increase requests from different apps may exceed the capacity of the network or service segment.

[0042] Indirect and implicit conflicts may include:

[0043] • Different apps may target different configuration profiles to optimize the same metric according to the app’s objective over the same network or service segment. Such an optimization may have uncontrollable or inadvertent impacts for overall network functioning.

[0044] • Different apps may target to optimize different, but dependent, metrics over the same network or service segment. Optimizing the different metrics may have implicit, unwanted, and adversary side effect for overall network functioning.

[0045] Embodiments of the invention identify and manage such conflicts. To appropriately manage conflicts, causes of conflicts (direct, indirect or implicit) need to be identified. Causes of conflicts may have conflicting priorities and associated trade-off from each apps’ perspective, thus a framework is needed to establish the trade-off managed environment. This is achieved by providing an adapted CAP theorem suitable for network infrastructure to establish the trade-off managed environment.

[0046] Embodiments of the invention can reduce the degree of severity of the conflicts by harmonizing the degree of changes introduced. In this way, the network infrastructure will not run into a panic mode abruptly due to sudden changes. This is achieved by introducing a Perturbative Harmonization Approach for introducing changes into the Network.

[0047] Embodiments of the invention use continuous learning to mature methodologies of harmonized changes in a trade-off managed environment through continuous feedback and rewards. Matured (through reinforcement learning) agents representing individual apps or intents can substantially reduce occurrences of conflicts with the help of guidance offered by a Central Critic Framework (explained below). This is achieved by introducing a Multi -Agent Reinforcement Learning (MARL) architecture suitable for addressing the specific needs of app hosting network infrastructure.

[0048] Figure 1 is a flow chart illustrating method steps in a conflict management method 100 for managing conflicts within a network infrastructure.

[0049] At step 102, method 100 comprises identifying a conflict based on: (i) at least one control action instigated by an intent of an application, and / or (ii) one or more environment observables of the network.

[0050] At step 104, method 100 comprises determining whether a nature of the conflict is direct, indirect or implicit based on the control action and / or the environment observables. For example, this step may be performed as described below in relation to Figures 6 and 7.

[0051] At step 106, method 100 comprises identifying one or more causes of the conflict based on the nature of the conflict, the control action and / or the environment observables. For example, this step may be performed as described below in relation to Figures 6 and 7. Steps 104 and 106 may be performed iteratively.

[0052] At step 108, method 100 comprises prioritising the one or more causes of the conflict based on a predetermined prioritisation framework. The predetermined prioritisation framework (also referred to herein as a “trade-off managed environment”) may be based on CAP theorem, more specifically, an adapted CAP theorem. This adapted CAP theorem may be an adapted CAP theorem for managing trade-off for app hosting network infrastructure. The trade-off managed environment and the adapted CAP theorem is discussed below in relation to Figure 2.

[0053] Steps 102-108 may be performed by the critic DNN described in more detail below. The critic DNN may work with other components of the central critic framework such as the agent monitor and the environment monitor. These components are described below in relation to Figure 3.

[0054] At step 110, method 100 comprises sending guidance to the application based on the conflict, the one or more causes of the conflict, and / or the priority of the one or more causes of the conflict, wherein the guidance uses a perturbative harmonization approach to guide changes into the network to reduce severity of the conflict. The perturbative harmonization approach is described in more detail below (see section titled “Perturbative Harmonization Approach”).

[0055] The guidance may comprise one or more of:

[0056] • changing a frequency of monitoring of the environment observables;

[0057] • changing the predefined time interval of the perturbative harmonization approach; • changing the direction of the control action;

[0058] • pausing the control action;

[0059] • delaying the control action; and

[0060] • withdrawing the control action.

[0061] The perturbative harmonization approach may comprise:

[0062] • actioning a first change;

[0063] • waiting for a predefined time interval;

[0064] • evaluating the effect of the first change;

[0065] • determining a direction of a second change, based on the effect of the first change;

[0066] and

[0067] • actioning a second change.

[0068] The method may be performed by a MARL model, for example, the MARL architecture described below in relation to Figure 3. The MARL model may comprise: (i) a plurality of agents, wherein the intent of an application is associated with an agent of the plurality of agents, and wherein each of the plurality of agents is associated with a corresponding intent; and (ii) a central module (referred to herein as a “central critic framework”) which communicates with each of the plurality of agents and monitors an environment of the network to obtain the environment observables (referred to herein as Obsrl []). The agent may comprise a learning agent configured to implement a Markov process for reinforcement learning for its corresponding intent. The agent may comprise a guidance agent configured to obtain central guidance from the central module and to obtain local guidance by communicating with the learning agent, wherein the guidance comprises the local guidance and the central guidance. The central module may comprise a deep neural network (referred to herein as the “critic DNN”) configured to implement a Markov process for reinforcement learning. The steps of determining the nature of the conflict (S.104) and identifying one or more causes of the conflict (S.106) may be performed using the deep neural network which is trained to recognise relationships between variables in an environment of the network infrastructure within a purview of the intent, as described in detail below under the heading “Central Critic Framework: The Critic Deep Neural Network”. The relationships between variables may comprise one or more of a relationship between:

[0069] • a plurality of environment observables (e.g. Obsrl [] described below) and a plurality of control actions (e.g. Ctrl [] described below), wherein the plurality of environment observables comprise the one or more environment observables, and wherein the plurality of control actions comprise the at least one control action;

[0070] • the plurality of control actions and a plurality of rewards (e.g. Rtdescribed below) wherein the plurality of rewards relate to the Markov process;

[0071] • the plurality of control actions and a plurality of rewards with penalties (e.g. Rt-with- penalty described below), wherein the plurality of rewards with penalties relate to the Markov process.

[0072] The relationships between variables comprise mappings of one variable against another. For example, for the relationship between environment observables and control actions, this may comprise mapping control actions against observables. This is described in relation to Figure 6 below.

[0073] If the conflict cannot be resolved using steps 102-110, method 100 may further comprise prompting a user to intervene (referred to herein as administrative intervention).

[0074] Trade-off Managed Environment

[0075] Figure 2 illustrates CAP theorem 200 and adapted CAP theorem 202. As shown in Figure 2, these can be illustrated as trade-off choice triangles. In system architecture, CAP theorem (or Brewer’s Theorem) states that any distributed system can provide only two of the following three guarantees for Consistency, Availability, and Partition Tolerance. I.e. it is possible to guarantee consistency and availability, consistency and partition tolerance, or availability and partition tolerance, but not all three at once. Embodiments of the invention adapt CAP theorem for network architecture trade-off management. In network architecture, adapted CAP theorem would imply that the distributed systems of network architecture can provide only two of the following three guarantees for Consistency & aligned functions (C’), Availability & aligned functions (A’), and Partition tolerance and aligned functions (P’).

[0076] The trade-off managed environment defines the adapted CAP theorem as relevant by associating network infrastructure KPIs to CAP specific observables, and by associating network control capabilities to CAP specific controllables. The trade-off managed environment defines acceptable threshold limits as well as degrees of severity for unacceptable threshold breaches corresponding to the measured observables (which may be caused by conflicts, but not necessarily). The trade-off managed environment also defines time window At as relevant periodicity for network infrastructure for monitoring observables. Depending on the scenario, At may be a few seconds, a few minutes, or more. The trade-off management also defines default prioritization principle of network infrastructure (e.g., C’A’ > C’P’ > A’P’) with options of overwrite by network administrators.

[0077] Adapted CAP theorem groups functions into:

[0078] (i) Consistency and aligned functions (C’)

[0079] (ii) Availability and aligned functions (A’)

[0080] (iii) Partition tolerance and aligned functions (P’)

[0081] For example, functions which fall under C’ may comprise one or more of the following:

[0082] • Consistency, Stability, and Reliability of Services

[0083] • Consistency of Quality of Services (QoS)

[0084] • Services & Resources Maintainability

[0085] • Scheduling Consistency

[0086] • Sustainability of Throughput / Latency / Serviceability KPIs

[0087] Functions which fall under A’ may comprise one or more of the following: • Availability of Services

[0088] • Resource Allocation & Admission Control

[0089] • Resource Deallocation & Termination Control

[0090] • Access Barring Control Function for Resources & Services

[0091] • Sustainability of Availability KPIs

[0092] Functions which fall under P’ may comprise one or more of the following:

[0093] • Redundancy and Scalability of Services

[0094] • Control of Node Addition / Modification

[0095] • Control of Cell and Slice Addition / Modification

[0096] • Dual Connectivity Control Function

[0097] • Carrier Aggregation Control Function

[0098] • Handover & Associated Control Function.

[0099] Each of these functions may be represented by observables (denoted as Obsrl []). These observables may be monitored for one or more of the following:

[0100] • Health & Performance Parameters

[0101] • Current Value of Configuration Elements / Configuration Profiles / Status of Policies

[0102] • Thresholds Breaches

[0103] • Degree of Severity corresponding to Degree of Thresholds Breaches.

[0104] Each of these functions may also be represented by controllables (denoted as Ctrl []). These controllables may be actioned for one or more of the following:

[0105] • Change of existing Configuration Values / Policies

[0106] • Introducing gradual degree of Change from Default Configurations / policies.

[0107] Perturbative Harmonization Approach

[0108] For introducing changes into a network in a harmonized way, a perturbative approach is proposed. In this approach, a control action A will be decomposed into a perturbative series of multiple smaller actions, and the complete series will add up to the entire “change” of configuration profile or policy the action A will target for:

[0109] A = Ao + £1A1 + 82A2 + 83A4 + 85A5 +... + 8nAn

[0110] Where n is the degree / order of perturbation and 8 is the co-efficient at a particular position in the series.

[0111] The order of perturbation will be decided dynamically based on the nature of action, the changes being triggered in the configuration profile or policy, and their perceived impact on the environment. The coefficients will be dependent on the nature of action, the corresponding changes, and the impact of a previous action. The coefficients will accordingly be used as adjustment factors.

[0112] Between action Anto An+i, there will be a time gap of At. During this time gap, an assessment will be carried out to determine whether the action in the subsequent order should progress towards an intended direction (e.g. j1) or should initiate a correction towards opposite direction (e.g. J,).

[0113] Thus, the series could look like this:

[0114] A = AoT+ 81A1T+ 82A2T+ 83A3T+ 84A4T+ 85A5t... + 8nAnt

[0115] Or it could look like this:

[0116] A = Aot+ 8IAI$+ 82A2T+ 83A3+ 84A4+ 85A5... + snAnt

[0117] By incorporating this time gap, there is a time break in the perturbative series which makes the change gradual. A change is made, then after a time break At, the effect of the change is evaluated, and the direction of the next change is then determined.

[0118] Multi-Agent Reinforcement Learning Architecture

[0119] Figure 3 shows a Multi-Agent Reinforcement Learning (MARL) architecture 300. The architecture 300 comprises a trade-off managed environment 302 within an app hosting network infrastructure 304. Apps 306abc are hosted on top of the app hosting network infrastructure. Each app or intent 306abc has its own individual actor or agent 308abc which provides individualised guidance for each app / intent. Each individual actor or agent 308 comprises a learning agent 310 and a guidance agent 312. Each individual actor or agent is in communication with a central critic framework 314 which provides central guidance to the individual actors / agents relating to the trade-off managed environment as a whole (i.e. “central guidance”). The central critic framework comprises an actor / agent monitor 316, an environment monitor 318, a central guidance module 320, a critic deep neural network (DNN) 322, and a replay buffer and database 324.

[0120] An app / intent 306 sends a control action to its learning agent 310. The agent 308 and App / intent 306 both monitor the environment 302 independently. The guidance agent 312 sends guidance to the app 306 as and when needed. The learning agent 310 sends the control action to the environment 302, and it sends the monitoring information to the Agent Monitor 316 component of central critic framework 314. The environment monitor 318 monitors the environment 302 independently and is in communication with the individual agents 308. The central guidance module 320 is in communication with each guidance agent 312 and sends guidance and receives confirmation on the use of guidance.

[0121] App / intent Function

[0122] Each app is in control of what it observes and controls and has an “end goal” which is what it wants to accomplish. The guidance provided to the app suggests what to control, e.g. suggesting making slow and gradual change dependent on how the situation is evaluated at each time break.

[0123] An intent is considered as a function that attempts to execute a single control action over the environment. An app can host multiple intents if it requires executing multiple control actions. In such a case, each intent will have its own individual actor / agent. Every intent aligns with an individual actor / agent within the trade-off managed environment of MARL architecture.

[0124] The app function monitors environment observables, represented by: Obsrl [Oc{i...n}, OA{i...n}, Op{i...n}], where:

[0125] Obsrl [] represents a set of observables;

[0126] Oc{i...n} represents observable parameters (1 to n) belonging to consistency aligned functions (as described above in relation to the adapted CAP theorem); OA{i...n} represents observable parameters (1 to n) belonging to availability aligned functions (as described above in relation to the adapted CAP theorem); and

[0127] OP{i...n} represents observable parameters (1 to n) belonging to partition tolerance aligned functions (as described above in relation to the adapted CAP theorem).

[0128] The app function receives guidance and acts accordingly. The app function may receive: guidance on perturbative decomposition of control action;

[0129] guidance on timeliness / periodicity to maintain for monitoring observables and for sequence of perturbative actions (i.e. At); and / or

[0130] guidance on the direction of control action, i.e. whether the control action should continue in a forward direction (t), or continue in the opposite direction (J,), or pause, or withdraw.

[0131] The app function may execute a control action, represented by: Ctrl [Ac{n}] OR Ctrl [ AA{N}] OR Ctrl [Ap{n}], where:

[0132] Ctrl [] represents a set of controllables;

[0133] Ac{n} represents a control action (nth) belonging to consistency aligned functions (as described above in relation to the adapted CAP theorem);

[0134] AArepresents a control action (nth) belonging to availability aligned functions (as described above in relation to the adapted CAP theorem); Aprepresents control action (nth) belonging to partition tolerance aligned functions (as described above in relation to the adapted CAP theorem).

[0135] Individual Actor / Agent

[0136] The individual actor / agent works with the app / intent function by passively intercepting the control actions. The individual actor / agent implements a Markov decision process (where S represents state, R represents reward and A represents action). Figure 4 shows the relationship between the agent 308 and its environment 402.

[0137] With reference to Figure 4, the environment’s current state S at time t is represented by Stas a gradient of the observables set with the same parameters as identified for target / goal with their directional indicators:

[0138] St = VObsrl [OC{nl}t t, OC{n2}t T, OA{nl}t $, OA{n2}t J,, C {n3}t J,] The end goal ST is represented by the same set of parameters for Stwith their final targeted value of final targeted condition:

[0139] ST = VObsrl [

[0140]

[0141] Oc{ni}T f, Oc; ri2; r T, OA{ni}T T, OA; n2; T J,, OP{n3}T J,]

[0142] Reward Rtis considered as a measure of the likelihood of achieving state ST from state St+i. Thus, Rt= P (ST | St+i) with discount factor y.

[0143] Action Atis represented by n (Ctrl [Ac |A |p{n}]) where n represents a policy and Ac |A |p

[0144]

[0145] represents an action taken on either consistency aligned or availability aligned, or partition tolerance aligned functions.

[0146] During initial handshaking between an app and its individual actor / agent, the individual actor / agent recognizes the Obsrl [] set, Ctrl [] set, and Obsrl-Target [] set of the app and keeps the learning agent and the guidance agent updated accordingly.

[0147] Learning Agent

[0148] The learning agent implements the overall Markov process for reinforcement learning corresponding to the particular intent / app. The learning agent maintains the ability to continue / delay / pause / stop control actions from reaching the app hosting networking infrastructure. The learning agent co-works with the guidance agent.

[0149] The learning agent is continuously learning. The learning agent takes an action, then assesses the reward after At to determine if the action moved the app towards or away from the end goal.

[0150] Guidance Agent

[0151] The guidance agent identifies local guidance by co-working with the learning agent. The guidance agent retrieves guidance from the central critic framework. The guidance agent notifies guidance to its corresponding app / intent. The guidance agent analyses the ongoing situation of states and guidance received, and in-case of criticality, works with the learning agent to delay / pause / stop the app-initiated control actions. The guidance agent also maintains a default configuration on possible degree / order of perturbation that can be introduced for the control action and a default guidance on timeliness / periodicity between subsequent order of perturbation and starts perturbative series of control action by keeping a target to complete the magnitude of change through control actions series well before max degree / order of perturbation. In other words, the guidance agent starts with default guidance.

[0152] By leveraging an epsilon greedy strategy, the guidance agent finds suitable opportunities to improvise the degree / order of perturbation from its default value. The guidance agent loads the trade-off managed environment’s criteria of threshold breaches beyond which the control action cannot be continued for self-processing.

[0153] An epsilon greedy strategy is a technique in reinforcement learning and decision-making used to balance exploration and exploitation. This strategy is applied when an agent needs to choose actions in an environment to maximize its total reward overtime.

[0154] As part of ongoing execution, the guidance agent maintains a map datastore to recognize what incremental control action n (Ctrl [Ac |A |p{n}t]) results into what change of observables set VObsrl [OC |A |P’{m}tTh |. The guidance agent also maintains a map datastore to recognize what incremental control action n (Ctrl [Ac |A |p{n}t]) results into what Rt.

[0155] By leveraging an epsilon greedy strategy, the guidance agent gradually starts recommending a magnitude of change and a directional indicator of control action n (Ctrl [Ac |A |p{n}tTD OR 7i (Ctrl [Ac |A |P{n}tJ,]) in the perturbation series through self-assessment and through the considerations of central critic framework. In other words, as the guidance agent learns and gains confidence, it moves away from its default guidance and starts making more tailored suggestions.

[0156] App / intent Onboarding and Leaming / Guidance Execution Flow

[0157] Figure 5 shows a flowchart 500 illustrating app / intent onboarding and learning and guidance execution flow. At step 502, the app registers its onboarding request, and provides info on its Obsrl [] set and Ctrl [] set to the app hosting network infrastructure. At step 504, the app registers its end goal or “unperturbed condition” by establishing target observables, Obsrl- target [], or by establishing a condition among the Obsrl [] parameters. Examples of establishing a target using conditions include:

[0158] • a (Oc{ni})+ b (OA{n2}) = c, where a and b are coefficients, and c is a constant;

[0159] • (Oc{ni}) = a (OP{n2})2+ b (OP{n2}) + c, where a and b are coefficients, and c is a constant;

[0160] • any other form of relational equation.

[0161] At step 506, if the app has been programmed to quit upon certain threshold breach conditions, the same will need to be registered with the app hosting network infrastructure. Examples of threshold breach conditions include:

[0162] • Oc’{nl}> 80% of (Oc’{nl}Fx;

[0163] • (Oc{ni}) « a (Op{n2})2+ b (Op{n2}) + c, where a and b are coefficients, and c is a constant;

[0164] • Any other form of relational equation

[0165] At step 508, upon completion of registration, the app hosting network infrastructure onboards the app. At step 510, as part of onboarding, the app gets positioned into the trade-off management choice matrix as per the default prioritization principle.

[0166] At step 512, the app starts functioning autonomously, monitoring Obsrl [] set and initiating Ctrl [] as needed and as per guidance received. At step 514, the app’s Ctrl [] is processed through MARL’s individual actor / agent and supervised by the central critic framework. At step 516, as necessary, local and / or global guidance is provided to steer the app’s functioning to trade-off managed and conflict managed situations.

[0167] As such, steps 502-508 are initiation steps to onboard the app. Steps 512-516 repeat as needed once the app is onboarded.

[0168] Central Critic Framework: The Actor / Agent Monitor

[0169] The actor / agent monitor implements a monitoring mechanism for the agent / actor specific Obsrl [] set and Ctrl [] set. These sets are environmental sets. The actor / agent monitor also implements a monitoring mechanism for agent / actor specific learning activities, guidance issuance, decision making, exploration coverage and exploitation coverage. The actor / agent monitor also aggregates individual actor / agent monitoring info at an environment level and co-works with the environment monitor and the guidance module.

[0170] Central Critic Framework: The Environment Monitor

[0171] The environment monitor implements a monitoring data consolidation mechanism for collecting and aggregating agent / actor specific Obsrl [] set into an environment wide Obsrl [] set, and by adding missing elements through direct monitoring actions towards the environment. The environment monitor also implements a monitoring mechanism to keep plotting the environment’s Obsrl [] set into the trade-off choice triangle (discussed above in relation to the adapted CAP theorem) and evaluates cluster-level outlying / topological distortions leading to compromised trade-off, as necessary. The environment monitor implements continuous evaluation of threshold breach conditions and raises these to the guidance module as and when they are detected.

[0172] Central Critic Framework: The Central Guidance Module

[0173] The central guidance module issues guidance on the magnitude and direction of the correction (t OR J,) of the control action 71 (Ctrl [Ac |A |p{n}t]) at a given time cycle (At) for a specific agent as per the current state. The central guidance module issues guidance on delay / pause / stop specific control actions during the handling of an exceptional situation (i.e. a situation with a severe fault in the environment or severe degradation of performance of the environment) or a threshold breach (refers to the trade-off managed environment’s criteria of threshold breaches). The central guidance module issues guidance cum notification to apps for cases where conflicts cannot be prioritized through trade-off criteria and requires administrative intervention, i.e. human intervention (refers to trade-off managed environment’s criteria for prioritization principles), and leverages the intervention as feedback for possible improvement of the central guidance module functionality. If the app’s target (ST) is not achieved after completion of the perturbative series of control action, the central guidance module notifies and guides app to seek administrative intervention (i.e. human intervention). Administrative intervention may be required in circumstances where a conflict cannot be managed by guidance, i.e. the conflict is beyond the capability of the system. By leveraging an epsilon greedy strategy, the central guidance module issues guidance on the order / degree of perturbation before initiating a perturbative series of control action by a specific agent.

[0174] Central Critic Framework: The Replay Buffer and Database

[0175] The replay buffer maintains the database of monitored information for each time cycle. The central critic framework maintains the replay buffer / experience of the state transition, action, and reward in the context of an environmental state and target. The central critic framework keeps records of guidance issued against each replay buffer / experience, its adoption by an individual agent, and corresponding action, if taken.

[0176] Central Critic Framework: The Critic Deep Neural Network

[0177] The critic DNN operates at aggregate level for all individual actors / agents and the overall environment. The DNN develops the relational strength of incremental control action n (Ctrl

[0178]

[0179] [Ac |A |p{n}t]) and its corresponding reward / result Rt for each specific definition of state Stagainst the purview of the agent and the overall environment. Upon completion of a perturbative series of a control action, the value of Rt-with-penalty is computed for each time cycle, factoring in the “penalty included reward” based on actual occurrence (i.e. a penalty is included if the action did not result in the desired result), then a relational strength matrix is developed for the incremental control action n (Ctrl [Ac |A |p{n}t]) and its corresponding Rt- with-penalty.

[0180] The critic DNN develops the relational strength between the gradient of an observable parameter VObsrl [Oc |A |p{ni}t TJ-] and the ongoing combinations of control actions it (Ctrl [Ac |A |p{n}t]) across all active agents. The critic DNN also improves the relational strength between the gradient of an observable parameter VObsrl

[0181]

[0182] [OC |A |P{ni}tTJ.] and various combinations of control actions witnessed during different time cycles, and builds a common minimum set of control actions that has most probable impact on the change of the observable parameter. The critic DNN plots the relational strength of each observable parameter VObsrl [Oc |A |P{ni t T-L | and associated individual control action n (Ctrl [Ac |A |p{n}t]) combination against the trade-off choice triangle and prepares a summarization relationship strength between each individual trade-off domain (C’, A’, P’, C’A’, C’P’, A’P’). For example:

[0183] • if observable parameter and control action both belong to C’ and with high relation strength, plot it closest to zC:

[0184] • if observable parameter and control action both belong to C’ and with medium relation strength, plot it little further from C’; and

[0185] • if observable parameter belongs to P’ and control action belong to C’, and with high relationship strength, plot it against C’P’ side.

[0186] Critic DNN’s Conflict Management Approach Flow

[0187] Figure 6 shows a flowchart 600 illustrating the critic DNN’s conflict management approach. At step 602, the DNN monitors aggregated info on individual actor / agents and overall environment. At step 604, the DNN evaluates the course of action to guide agents. The course can be classified as either a normal or an exceptional handling course. If the observables do not show any exceptional situation (fault / degraded performance) or threshold breach the situation will be classified as normal. If the observables show the situation as exceptional or threshold breached it will be treated as an exceptional course.

[0188] If classified as a normal handling course, the method proceeds to step 606. At step 606, each concurrently triggered Ctrl [Ac |A |p{n}t] f or J, is mapped against Obsrl [Oc |A |p{ni}t TJ.]. At step 608, the DNN correlates the aggregated mapping of each Obsrl [OC |A |P

[0189]

[0190] {ni}tor J,] against all witnessed combinations of actions Ctrl [Ac |A |P{n}t] f or J,]. At step 610, if the app’s target is not achieved after completion of the perturbative series of control action, the app is notified and prompted to seek administrative intervention (i.e. human intervention).

[0191] On the other hand, if classified as an exceptional handling course, the method proceeds to step 612. At step 612, the cause of exception or threshold breach is identified (e.g., Oc{ni} is exceptionally high). At step 614, all ongoing control actions that could impact the observable parameter are consolidated and the map (as described in relation to step 608) is developed (e.g., Oc{ni} is high since Ctrl [Ac{n}t] & Ctrl [Ap{n}t] is high. At step 616, for the identified map, the Direct Relation Map, Indirect Relation Map and Implicit Relation Map are discovered and classified (as described below).

[0192] If direct relation mappings are obtained, the method proceeds to step 618. At step 618, guidance is issued to all corresponding apps to consider changing the direction of control actions in the next order of perturbative correction. At step 620, after a time cycle, it is evaluated whether the observable parameter’s direction of change has started reversing. If yes, continue until a safe level is reached. If not, initiate delay / pause / stop for specific control actions as per trade-off configuration and invoke processing of indirect causes.

[0193] For all indirect relation mappings obtained, the method proceeds to step 622. At step 622, guidance is issued to all corresponding apps to consider changing the direction of control actions during the next order of perturbative correction. At step 624, after a time cycle, it is evaluated whether the observable parameter’s direction of change has started reversing. If yes, continue until a safe level is reached. If not, initiate delay / pause / stop for specific control actions as per trade-off configuration and invoke processing of implicit causes.

[0194] For all implicit relation mappings obtained, the method proceeds to step 626. At step 626, guidance is issued to all corresponding apps to consider changing the direction of control actions during the next order of perturbative correction. At step 628, after a time cycle, it is evaluated whether the observable parameter’s direction of change has started reversing. If yes, continue until a safe level is reached. If not, initiate delay / pause / stop for specific control actions as per trade-off configuration and notify apps, prompting them to seek admin intervention.

[0195]

[0196] Discovery

[0197] Figure 7 shows how the critic DNN 322, the agent monitor 316, the environment monitor 318 and the replay buffer and database 324 interact with one another within the central critic framework 314.

[0198] If a specific observable parameter has been registered by the apps to be monitored as the state and target set element, and specific registered control actions have been registered to introduce change, this combination of data elements of all different actively running apps maintained in a map will be considered as the model of “Direction Relations”.

[0199] If the critic DNN has identified a change of a specific observable parameter being dependent on concurrent control actions that are not already part of the direct relations map, the map of all such control actions being executed by different actively running apps and the specific observable parameter (along with directional indicators) will be considered as the model of “Indirection Relations”.

[0200] If a specific observable parameter is registered by an app as an element of state and target set and belonging to a trade-off domain (C’A7CP7A’P’) that is lower in priority for the trade-off domain of ongoing concurrent control actions invoked by different apps, and if those combinations of observable and controllable are not already part of direct and indirect relation maps, the entries of all such control actions and the specific observable parameter (along with directional indicators) will be added into indirect relations map and will be considered as extended model of “Indirection Relations”.

[0201] As one example, an observable parameter may be part of the partition tolerance aligned functions, and the prioritization principle may be C’A’ > C’P’ > A’P’. Thus, all ongoing control actions of consistency and availability aligned functions will be considered for pairing up with an observable parameter to form an indirect relations map.

[0202] If a specific observable parameter does not belong to the state and target set definition on any of the currently active apps and belongs to a trade-off domain that is lower in priority for the trade-off domain of ongoing concurrent control actions invoked by different apps, the entries of all such control actions and the specific observable parameter (along with directional indicators) will be added into a map and will be considered as model of “Implicit Relations”.

[0203] Figure 8 is a block diagram of an information processing apparatus 800 or a computing device 800, such as a data storage server, which embodies the present invention, and which may be used to implement some or all of the operations of a method embodying the present invention, and perform some or all of the tasks of apparatus of an embodiment. The computing device 800 may be used to implement any of the method steps described above, e.g. any of steps S102-110 and / or any processes described above. The computing device 800 comprises a processor 802 and memory 804. Optionally, the computing device also includes a network interface 806 for communication with other such computing devices, for example with other computing devices of invention embodiments. Optionally, the computing device also includes one or more input mechanisms such as keyboard and mouse 808, and a display unit such as one or more monitors 810. These elements may facilitate user interaction. The components are connectable to one another via a bus 812.

[0204] The memory 804 may include a computer readable medium, which term may refer to a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data accessible by and causing a computer (e.g., one or more processors) to perform one or more functions or operations. For example, the computer-executable instructions may include those instructions for implementing a method disclosed herein, or any method steps disclosed herein, e.g. any of steps S102-110, and / or any processes described above. Thus, the term “computer-readable storage medium” may also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the method steps of the present disclosure. The term “computer-readable storage medium” may accordingly be taken to include, but not be limited to, solid-state memories, optical media and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media, including Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid state memory devices).

[0205] The processor 802 is configured to control the computing device and execute processing operations, for example executing computer program code stored in the memory 804 to implement any of the method steps described herein. The memory 804 stores data being read and written by the processor 802 and may store data and / or programs for executing any of the method steps and / or processes described above. As referred to herein, a processor may include one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. The processor may include a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processor may also include one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. In one or more embodiments, a processor is configured to execute instructions for performing the operations and operations discussed herein.

[0206] The display unit 810 may display a representation of data stored by the computing device, such as GUI windows and / or interactive representations enabling a user to interact with the apparatus 800 by e.g. drag and drop or selection interaction, and / or any other output described above, and may also display a cursor and dialog boxes and screens enabling interaction between a user and the programs and data stored on the computing device. The input mechanisms 808 may enable a user to input data and instructions to the computing device.

[0207] The network interface (network I / F) 806 may be connected to a network, such as the Internet, and is connectable to other such computing devices via the network. The network I / F 806 may control data input / output from / to other apparatus via the network. Other peripheral devices such as microphone, speakers, printer, power supply unit, fan, case, scanner, trackerball etc may be included in the computing device.

[0208] Methods embodying the present invention may be carried out on a computing device / apparatus 800 such as that illustrated in Figure 8. Such a computing device need not have every component illustrated in Figure 8 and may be composed of a subset of those components. For example, the apparatus 800 may comprise the processor 802 and the memory 804 connected to the processor 802. Or the apparatus 800 may comprise the processor 802, the memory 804 connected to the processor 802, and the display 810. A method embodying the present invention may be carried out by a single computing device in communication with one or more data storage servers via a network. The computing device may be a data storage itself storing at least a portion of the data. A method embodying the present invention may be carried out by a plurality of computing devices operating in cooperation with one another. One or more of the plurality of computing devices may be a data storage server storing at least a portion of the data.

[0209] The invention may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention may be implemented as a computer program or computer program product, i.e., a computer program tangibly embodied in a non-transitory information carrier, e.g., in a machine-readable storage device, or in a propagated signal, for execution by, or to control the operation of, one or more hardware modules.

[0210] A computer program may be in the form of a stand-alone program, a computer program portion or more than one computer program and may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules at one site or distributed across multiple sites and interconnected by a communication network.

[0211] Method steps of the invention may be performed by one or more programmable processors executing a computer program to perform functions of the invention by operating on input data and generating output. Apparatus of the invention may be implemented as programmed hardware or as special purpose logic circuitry, including e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0212] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memory devices for storing instructions and data.

[0213] The above-described embodiments of the present invention may advantageously be used independently of any other of the embodiments or in any feasible combination with one or more others of the embodiments. While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel methods and apparatuses described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of methods and apparatus described herein may be made.

Claims

We Claim:

1. A conflict management method for managing conflicts within a network infrastructure, the method comprising:identifying a conflict based on: (i) at least one control action instigated by an intent of an application, and / or (ii) one or more environment observables of the network;determining whether a nature of the conflict is direct, indirect or implicit based on the control action and / or the environment observables;identifying one or more causes of the conflict based on the nature of the conflict, the control action and / or the environment observables;prioritising the one or more causes of the conflict based on a predetermined prioritisation framework; andsending guidance to the application based on the conflict, the one or more causes of the conflict, and / or the priority of the one or more causes of the conflict, wherein the guidance uses a perturbative harmonization approach to guide changes into the network to reduce severity of the conflict.

2. The method of claim 1, wherein the method steps are performed by a multi -agent reinforcement learning, MARL, model comprising:a plurality of agents, wherein the intent of the application is associated with an agent of the plurality of agents, and wherein each of the plurality of agents is associated with a corresponding intent; anda central module which communicates with each of the plurality of agents and monitors an environment of the network to obtain the environment observables.

3. The method of claim 2, wherein the agent comprises a learning agent configured to implement a Markov process for reinforcement learning for the corresponding intent.

4. The method of claim 3, wherein the agent comprises a guidance agent configured to obtain central guidance from the central module and to obtain local guidance by communicating with the learning agent, wherein the guidance comprises the local guidance and the central guidance.

5. The method of any of claims 2 to 4, wherein the central module comprises a deep neural network configured to implement a Markov process for reinforcement learning.

6. The method of claim 5, wherein the steps of determining the nature of the conflict and identifying one or more causes of the conflict are performed using the deep neural network which is trained to recognise relationships between variables in an environment of the network infrastructure within a purview of the intent.

7. The method of claim 6, wherein the relationships between variables comprise one or more of a relationship between:(i) a plurality of environment observables and a plurality of control actions, wherein the plurality of environment observables comprise the one or more environment observables, and wherein the plurality of control actions comprise the at least one control action;(ii) the plurality of control actions and a plurality of rewards, wherein the plurality of rewards relate to the Markov process;(iii) the plurality of control actions and a plurality of rewards with penalties, wherein the plurality of rewards with penalties relate to the Markov process.

8. The method of any of the preceding claims, wherein the guidance comprises one or more of:changing a frequency of monitoring of the environment observables;changing the predefined time interval of the perturbative harmonization approach; changing the direction of the control action;pausing the control action;delaying the control action; andwithdrawing the control action.

9. A computer program which, when run on a computer, causes the computer to carry out the method of any of the preceding claims.

10. An information processing apparatus comprising a memory and a processor connected to the memory, wherein the processor is configured to perform a method, the method of any of the preceding claims.

Images