Method and apparatus for performing explainable reinforcement learning in a communication network

Abstract

The disclosure provides methods and apparatus for performing explainable reinforcement learning, RL, XRL, in a communication network (700) The method comprises obtaining (101) a request indicating a type of analysis to be performed on an RL model configured to perform a task associated with one or more communication network cells. The method comprises performing (103) the requested type of analysis, using analysis information, with respect to at least one cell of the one or more communication network cells. The method comprises compiling (104) an explanation information based on analysis results obtained by performing the requested type of analysis with respect to the at least one cell of the one or more communication network cells. The method comprises updating (105) the RL model based on the compiled explanation information.

Description

[0001] METHOD AND APPARATUS FOR PERFORMING EXPLAINABLE REINFORCEMENT LEARNING IN A COMMUNICATION NETWORK TECHNICAL FIELD

[0002] The disclosure relates to a method, apparatus, computer program and computer program product for performing explainable reinforcement learning, XRL, in a communication network.

[0003] BACKGROUND

[0004] In telecommunication, reinforcement learning, RL, optimizes various aspects of network management, from resource allocation to traffic management and beyond. RL is a fundamental sequential decision-making framework. At its core, RL is based on the concept of an agent learning by direct interaction with an environment. In simple terms, states represent configurations of the environment, and encapsulate all relevant information necessary for decision-making; actions denote the possible decisions that the agent can take while in each state; transition probabilities capture the environment's dynamics, specifying the probability of transitioning from one state to another upon taking a particular action; rewards serve as numerical feedback signals provided by the environment to the agent. In other words, rewards quantify the desirability of the agent's actions, guiding its decision-making process towards achieving predefined objectives. Explainable reinforcement learning, XRL, methods have been developed to analyse the behaviour of RL agents. In XRL methods, an explanation of an RL agent’s behaviour can generate additional information about several elements associated with the RL agent, such as input states, rewards, actions, or internal properties of the agent.

[0005] In the domain of radio network optimization, use of black-box machine learning, ML, models for RL brings challenges related to lack of trust and reliability in the outcome of these ML models, often due to limited or no transparency of the internal logic applied to achieve the outcome. Some basic statistical measures have been applied as initial attempts to understand and analyze the internal working of the black-box ML models utilized for RL. However, the existing techniques are insufficient to establish a sound and complete understanding of the ML-based outcome in the domain of radio networks optimization. So, there is clearly a need for enhanced, sophisticated mechanisms ofXRL to be applied to the existing solutions for making the underlying ML systems transparent in a true sense. There is a need to bring in radio domain expertise to complement the outcome of ML models. This is done for the purpose of validating the outcomes from ML models. Since the ML models remain to be black-boxed, this ends up being a manual process, mainly based on making guesses / speculations. The outcome of ML based operations in the radio network optimization and improvement are manually defined as goal-based inferences based on domain knowledge. The existing solutions [1] and PCT / EP2024 / 054704 generate explanations for the entire state-space on an aggregated basis (global explanations) or a very specific state (local level). The existing solutions [1] and PCT / EP2024 / 054704 do not provide granular explanations and often fail and results in answers such as “I cannot tell why agent takes an action”, when applied to the large-scale dataset.

[0006] SUMMARY

[0007] An object of the invention is to enable a more granular explainable reinforcement learning, RL, XRL in a communication network.

[0008] In a first aspect, a computer-implemented method for performing explainable reinforcement learning, RL, (XRL) in a communication network is presented. The method performed by a node in the communication network. The method comprises obtaining a request indicating a type of analysis to be performed on an RL model configured to perform a task associated with one or more communication network cells in the communication network. The method comprises performing the requested type of analysis, using analysis information, with respect to at least one cell of the one or more communication network cells. The method comprises compiling an explanation information based on analysis results obtained by performing the requested type of analysis with respect to the at least one cell of the one or more communication network cells. The method comprises updating the RL model based on the compiled explanation information. Hereby, this enables a way to perform XRL with respect to communication network cells and improve the RL model based on the results obtained using XRL.

[0009] According to a second aspect, there is presented a node in a communication network performing XRL in a communication network. The node comprising processing circuitry, a memory comprising instructions executable by the processing circuity suchthat the node is operable to obtain a request indicating a type of analysis to be performed on an RL model configured to perform a task associated with one or more communication network cells in the communication network. The node is operable to perform the requested type of analysis, using analysis information, with respect to at least one cell of the one or more communication network cells. The node is operable to compile an explanation information based on analysis results obtained by performing the requested type of analysis with respect to the at least one cell of the one or more communication network cells. The node is operable to update the RL model based on the compiled explanation information.

[0010] According to a third aspect, there is presented a computer program comprising instructions which when executed on a processor of the node, causes the node to perform a method according to the first aspect or any of its embodiments.

[0011] According to a fourth aspect, there is presented a computer program product which comprises a computer readable storage medium on which a computer program according to the second aspect is stored.

[0012] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0013] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, action, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, action, etc., unless explicitly stated otherwise. The actions of any method disclosed herein.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] A more complete understanding of the present embodiments, and the attendant advantages and features thereof, will be more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:

[0016] Figure 1 depicts a computer-implemented method for performing explainable reinforcement learning, RL, XRL, in a communication network.Figure 2 illustrates reward decomposition, RD.

[0017] Figure 3 illustrates autonomous policy explanation, APE.

[0018] Figure 4 depicts a process for generating explanation information corresponding to goal-wise analysis or high-level summary or level I summary (S1).

[0019] Figure 5 depicts a process for generating explanation information corresponding to action-wise analysis or action-wise summary or level II summary (S2).

[0020] Figure 6 depicts a process for generating explanation information corresponding to state-wise analysis or state-wise summary or level III summary (S3).

[0021] Figure 7 depicts a node in a communication network configured to perform a computer-implemented method for performing XRL in the communication network. DETAILED DESCRIPTION

[0022] The proposed disclosure teaches a skilled person to perform explainable reinforcement learning, XRL, on a cell-level basis by analyzing three major aspects-states, actions and goals, associated RL, with respect to each cell in a communication network. The proposed disclosure relates to the radio access network, RAN, functions and applications, specific to the machine learning, ML-based optimization problems, incorporating RL agents to control network parameters with the goal of optimizing certain network key performance indicators, KPIs.

[0023] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and viceversa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0024] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject-matter disclosed herein, the disclosed subject-matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject-matter to those skilled in the art.

[0025] Figure 1 depicts a computer-implemented method 100 for performing XRL according to at least a first embodiment of the disclosure. Figure 7 depicts a node 701 in a communication network 700 configured to perform the computer-implemented method 100 for performing XRL, according to at least a first embodiment of the disclosure. The method 100 performed by a node 701 in the communication network 700. The method comprises obtaining 101 a request indicating a type of analysis to be performed on an RL model configured to perform a task associated with one or more communication network cells in the communication network 700. The method 100 comprises performing 103 the requested type of analysis, using analysis information, with respect to at least one cell of the one or more communication network cells. The method 100 comprises compiling 104 an explanation information based on analysis results obtained by performing the requested type of analysis with respect to the at least one cell of the one or more communication network cells. The method comprises updating 105 the RL model based on the compiled explanation information. The RL model may be a trained RL model.

[0026] In a second embodiment of the disclosure, the method 100 of the first embodiment further comprises obtaining, 102, the analysis information for the type of analysis to be performed. According to the second embodiment the obtained request, 101, further indicates performing the requested type of analysis periodically. According to the second embodiment, the obtained request, 101, may further indicate two or more types of analysis to be performed. According to the second embodiment, performing 103 the requested type of analysis comprises performing the requested type of analysis with respect to each cell of the one or more communication network cells. According to the second embodiment, compiling 104 the explanation summarycomprises compiling the explanation summary based on analysis results obtained by performing the requested type of analysis with respect to each cell of the one or more communication network cells.

[0027] In method 100, according to the first and second embodiments, the task that the RL model is configured to perform may comprise radio network optimization. Radio network optimization is a complex task that involves configuring and managing network parameters to ensure efficient and reliable communication. RL is a type of machine learning, ML, where an RL agent learns to make decisions by interacting with an environment to achieve a specific goal. The agent receives feedback in the form of rewards or penalties and aims to maximize cumulative rewards over time. In method 100, according to the first and second embodiments, the RL model is a trained to perform a task associated with one or more communication network cells in the communication network.

[0028] For example, a setup of the RL model trained to perform a radio network optimization problem / task associated with one or more communication network cells in the communication network is described below:

[0029] Agent and Environment:

[0030] The radio network optimization problem can be framed as an RL problem where the RL agent represents the network controller, and the environment represents the radio network part of the communication network 700.

[0031] State, Action, and Reward:

[0032] State: Represents a current condition of the radio network (that is, the environment), such as channel conditions, user demand, interference levels, etc.

[0033] Action: Represent configuration changes the agent can make to optimize the radio network, such as adjusting power levels, reallocating resources, or changing frequency bands.

[0034] Reward: Represents a performance feedback based on the action taken, such as throughput, latency, energy efficiency, or Quality of Service, QoS, metrics.Examples of radio network optimization task comprises dynamic spectrum management, power control, resource allocation, handover management, load balancing, etc.

[0035] In method 100, according to the first and second embodiments, the RL model can be trained to perform ML Safety Shield, MLSS, based radio network optimization, as disclosed in WO 2022 / 023218 A1 and WO 2023 / 222188 A1. Example of MLSS based radio network optimization comprises MLSS- remote electrical tilt, MLSS-RET of an antenna.

[0036] In method 100, according to the first and second embodiments, performing the requested type of analysis may comprise performing reward decomposition, RD. Performing the requested type of analysis may comprise performing autonomous policy explanation, APE. Performing the requested type of analysis comprises performing RD and APE.

[0037] Figure 2 depicts RD and Figure 3 depicts APE. RD and APE are two XRL methods utilized in this disclosure for performing XRL in the communication network 700 and are discussed below:

[0038] RD

[0039] An ML model utilized for RL predicts an action to be performed by the RL agent at time instant t + 1 based on an observation for time instant t comprising one or more values associated with the state of the environment at time instant t.

[0040] As shown in Figure 2, the observation at t + 1 and the action at t + 1 predicted by the ML model and performed by the RL agent are provided as inputs 201 for the process of RD 202. The action performed by the RL agent is provided with a reward 204 by a reward function 203. The reward function 203 is decomposed into one or more reward components (reward 1 and reward 2, as shown in Figure 2). In the RD 202 process, the ML model is decomposed into one or more branches (ML model- branch 1, ML model- branch 2, as shown in Figure 2). Each branch of the decomposed ML model is associated with one reward component.

[0041] Example:Formulate reward function (r_f), based on network KPIs (such as, balance coverage, improve performance) and other parameters p, for an action taken at time step t, as follows:

[0042]

[0043] For example, the following reward function comprises two reward components- reward component 1 corresponds to a goal / KPI ‘Balance Coverage’ and reward component 2 corresponds to the goal / KPI ‘Improve performance’.

[0044] Reward function

[0045] gapt ~ gapt+1)(l - covft) + weight

[0046] • (max(covD0Ft, qualDOFf) — max (covDOFt+1, qualD0Ft+1y) Reward component 1:

[0047] (gapt ~ gapt+1) l - covft)

[0048] Reward component 2:

[0049] weight • (max(covD0Ft, qualDOFf) — max (covDOFt+1, qualD0Ft+1y)

[0050] Terms of the reward equation: gapt+1, covD0Ft+1, qualD0Ft+1are observations for time instant t + 1 as a result of an action predicted by the ML model for time instant t + 1. Terms of the reward equation: gapt, covD0Ft, qualD0Ftare observations at time instant t for the ML model to predict the action at time instant t + 1.

[0051] An objective of the RD 202 process is to decompose a complex, aggregated reward function 203 into simpler, more interpretable components that represent distinct aspects of the environment or the task. RD as illustrated in Figure 2 and as discussed in [2] decomposes rewards into sums of semantically meaningful reward types, so that actions can be compared in terms of trade-offs among the types. RD aims to provide explanations for each action of RL by analysing the rewards associated with the action.

[0052] APE APE provides explanation for RL outcomes in the form of natural language clauses to relevant queries through analysis of a target state space of the RL model, specific to a selected target action. The queries could be presented in one of the following three types of formats-Type I query: When do you do <action>?

[0053] Type II query: Why didn’t you do <action> in state <state #>?

[0054] Type III: What will you do when <conditions>?

[0055] Type I query (When do you do <action>?) address the explainability requirements on a global level, considering all the states in the action space for the RL agent. The answer is generated through analysing the most frequent states in the target state space for a given <action> in a query. Considering the MLSS-RET use case, facilitating the customer need for intelligent tilt change of an antenna, the relevant <action> would be one of the following - ‘Uptilt’ referring to increase in the tilt, ‘Make No Change’ 'corresponding to no change in tilt and ‘Downtilt’ referring to decrease in tilt. Figure 3 shows a process for extracting granular explanations at cell level for the radio network, presented as conditional clauses with frequency distribution.

[0056] Type II query (Why didn’t you do <action> in state <state #>?) provides contrastive explanations at a local level, concerning a specific contrastive action that was not taken for a selected state in the state space. For example, the type II query can be formulated as “Why RL agent didn’t perform ‘down tilt’ when ‘coverage is good but with high interference’?” In this case, the action refers to ‘down tilt’ and the state refers to ‘coverage is good but with high interference’.

[0057] Type III query (Type III: What will you do when <conditions>?), provides explanations through analysing a situational behaviour of the RL agent and identifying the most likely actions (as percentage values) satisfying the conditions. For example, the type II query can be formulated as “What will the RL agent do when ‘coverage is good but with high interference’?” In this case, the state refers to ‘coverage is good but with high interference’.

[0058] RD and APE are typically executed in combination to perform different types of analysis, on the RL model, such as a goal-wise analysis, a state-wise analysis and an action-wise analysis.

[0059] In the first and the second embodiments, the type of analysis comprises one of the goal-wise analysis, state-wise analysis and action-wise analysis.Goal-wise analysis in the context of XRL involves understanding how the RL agent's decision-making process is influenced by specific goals or objectives. In RL, goals can be explicit targets (for example, KPIs of the communication network) or desired outcomes that the agent aims to achieve through its interactions with the environment (for example, the communication network). Goal-wise analysis focuses on elucidating how these goals shape the agent's strategies, policies, and actions.

[0060] Action-wise analysis in the context of XRL refers to an examination and interpretation of individual actions taken by the RL agent within its environment (for example, communication environment). In RL, the agent learns to make decisions by interacting with its environment, aiming to maximize cumulative rewards overtime. Understanding why the agent chose specific actions in particular states is crucial for the interpretability and trustworthiness of the decision-making process.

[0061] State-wise analysis in the context of XRL focuses on understanding and interpreting how the RL agent perceives and evaluates different states within its environment (for example, communication environment). This analysis aims to provide insights into the agent's decision-making process by examining how states influence the selection of actions and the pursuit of goals.

[0062] The explanation information comprises three different levels of explanation information. Each level of explanation information corresponds to one type of analysis (example, goal-wise analysis, state-wise analysis and action-wise analysis). Explanation information corresponding to goal-wise analysis is also referred to as high-level summary or level I summary and is denoted by S1. Explanation information corresponding to action-wise analysis is also referred to as action-wise summary or level II summary and is denoted by S2. Explanation information corresponding to state-wise analysis is also referred to as state-wise summary or level III summary and is denoted by S3.

[0063] The explanation information is extracted using the following two modules:

[0064] (i) Automated Goal Analysis, AGA, provides explanation information by determining whether specific goals, for example, radio network coverage or performance, have been fulfilled or not.

[0065] (ii) Action-wise set of Conditional Clauses, ACC, provides a set of clauses pertaining to an action and determines respective distribution of frequenciesfor each action in a set of actions. This module is a variant of the existing algorithm for APE. ACC identifies conditions associated with specific actions. The following modifications are proposed to existing algorithm of APE:

[0066] a. provide conditional clauses for all the different actions in a single step, presented in easy-to-scale, compact tabular format, with each row containing a summary associated with a specific cell of the communication network 700;

[0067] b. compute minimized set of target states based on frequency analysis, instead of Quine-McCluskey, QM, method as given in existing solutions; c. conditional clauses are determined by evaluating the KPI values for denser regions of range values (facilitated with multiple thresholds per KPI). The existing solution [1] evaluates the KPI with respect to a single threshold value;

[0068] d. explanation information is generated for a set of states belonging to a specific cell in the communication network, thus, offering granularity for extracting explanations at cell-level basis. In existing solutions explanation is determined for aggregated state-space or a single state whereas in the proposed solution explanation information is generated at cell-level.

[0069] The cell-level explanation information is made available to the user at scale for the live-network, over a period (e.g., daily or 10-days).

[0070] The explanations are updated periodically by analysing new operational data collected from the live network. The high-level summary can be further complemented with action-wise summary for specific actions that do not satisfy a goal. The high-level summary with the action-wise summary for specific actions that do not satisfy a goal can be provided for a specific cell.

[0071] Given an RL-based optimization problem in the RAN domain, to optimize certain network KPIs by controlling one or more network parameters, the proposed solution aims to extend the RL-based optimization solution through facilitating explanation information (presented as explanation information per cell) at scale for live-network. The proposed solution aims to extend the RL-based optimization solution throughfacilitating explanation information (presented as explanation information per cell) at scale for live-network, such that the explanation information is periodically updatable. Figure 4 depicts a process for generating explanation information corresponding to goal-wise analysis or high-level summary or level I summary (S1 ) according to the first and second embodiments of the disclosure. Level I summary is generated using the two modules, AGA and ACC. Node 701 executes modules AGA 400d and ACC 400e. The first module, AGA determines whether a set of goals (prespecified, using domain expertise) have been met or not. For example, in MLSS based radio network optimization use case, the goals to be met by the RL agent may comprise:

[0072] (i) Balance coverage of the radio network,

[0073] (ii) Improve performance of the radio network, and

[0074] (iii) Propose no change to the radio network

[0075] The goals are modelled through KPIs of the communication network 700.

[0076] As shown in Figure 4, a user 400a may transmit, 401, to the node 701 via a dashboard 400b, the request indicating the type of analysis to be performed on the RL model configured to perform a task associated with one or more communication network cells in the communication network 700. The dashboard 400b serves as an interface between the user 400a and the node 701. The node 701 obtains the request indicating the type of analysis to be performed on the RL model configured to perform a task associated with one or more communication network cells in the communication network 700. The type of analysis comprises goal-wise analysis.

[0077] The node 701 may comprise the RL model. The RL model may be comprised in another node of the communication network 700. The node 701 may execute the RL model. The RL model may be executed in another node of the communication network 700. The node 701 utilizes analysis information in order to perform the requested type of analysis. According to the first embodiment, the AGA 400d module of the node 701 may comprise the analysis information. According to the second embodiment, the AGA 400d module of the node 701 may obtain the analysis information from a database 400c. The node 701 may be associated with the database 400c. The analysis information comprises the RL model and communication network data used for training the RL model.The node 701 performs the requested type of analysis, using the analysis information, with respect to at least one cell of the one or more communication network cells. Performing the requested type of analysis comprises performing RD with respect to at least one cell of the one or more communication network cells. Performing RD with respect to at least one cell of the one or more communication network cells comprises identifying, 404, reward components for each cell in the one or more communication network cells. Performing RD with respect to at least one cell of the one or more communication network cells comprises computing, 405, reward composition for each cell in the one or more communication network cells. Performing RD with respect to at least one cell of the one or more communication network cells comprises generating, 406, goal summary for each cell in the one or more communication network cells. RD with respect to at least one cell of the one or more communication network cells is performed by the AGA module 400d of node 701. Performing RD with respect to a cell of the communication network 700 enables the determination of whether a goal has been satisfied by the RL model in a cell or not. The goal summary comprises identifiers, for example Cell # 001... Cell # 00X, of one or more communication network cells with respect to which RD was performed. An example goal summary is presented in Table 1. Each cell, identified by an identifier, is associated with two values for each goal- a first value indicating a number of times the goal was satisfied in the cell and a second value indicating a number of times the goal was not satisfied in the cell by the RL model. The first value and second value are integers, respectively. For example, a cell identifier denoted by Cell #001 has Goal #1 being satisfied by the RL model #count_true times and Goal #1 being not satisfied by the RL model #count_false times.Table 1: Outcome of AGA- Goal summary

[0078] Goal Goal #1 Goal # 1 .... Goal # Q Goal # Q Goal met True False ... True False

[0079] Cell Id

[0080] Cell # 001 #count_true #count_false #count_true #count_false Cell # 002

[0081] Cell # 00X #count_true #count_false #count_true #count_false

[0082]

[0083] RD is performed by the node 701 after an action, proposed by the RL model, has been performed in the cell of the communication network 700.

[0084] The goal summary (as shown in Table 1) is sent, 407, to the dashboard 400c in order to make it available for the user 400a.

[0085] The goal summary (as shown in Table 1) is sent, 408, to the second module ACC for complementing the goal summary with conditional clauses for each cell (identified using the identifier “Cell Id”).

[0086] According to the first embodiment, the ACC 400e module of the node 701 may comprise the analysis information. According to the second embodiment, the ACC 400e module of the node 701 may obtain, 409, the analysis information from a database 400c. The node 701 may be associated with the database 400c. The analysis information comprises the RL model, communication network data used for training the RL model, states associated with the RL model, a set of actions proposed by the RL model during RL model execution in the communication network environment and predicates associated with the RL model. Predicates associated with the RL model defines conditions or properties of states, actions, or entities within the communication network environment.

[0087] The node 701 performs the requested type of analysis, using the analysis information, with respect to at least one cell of the one or more communication network cells. Performing the requested type of analysis comprises performing APE with respect to at least one cell of the one or more communication network cells. APE with respect toat least one cell of the one or more communication network cells is performed by the ACC module 400d of node 701. Performing APE with respect to at least one cell of the one or more communication network cells comprises initiating, 410, Type 1 query “When does the RL agent propose <action> from the set of actions?” for each cell in the one or more communication network cells. For each cell in the one or more communication network cells, the ACC module 400e of node 701 identifies, 411, target states for each <action> in the set of actions proposed by the RL model during RL model execution in the communication network environment. For each cell in the one or more communication network cells, the ACC module 400e of node 701 generates, 412, conditional clauses for each <action> in the set of actions proposed by the RL model during RL model execution in the communication network environment. Target states refers to a set of states, associated with the RL model, causing the RL model to propose the <action> during execution. Conditional clauses are generated by analysing the most frequent states in the target states causing the RL model to propose the <action> in the query. Outcome of ACC module 400e is presented in Table 2. Outcome of ACC 400e module is the conditional clauses for each cell in the one or more communication cells.

[0088] For example, in table 2, Cell #001 performed Action 1 with a frequency of 60% when condition 1a is satisfied. Condition 1a is defined by Low KPI1, Medium KPI2, High KPI 3. In this example, Cell #001 is the cell identifier. Condition 1a is a conditional clause. Low KPI1, Medium KPI2, High KPI 3 are the target states.Table 2: Outcome of ACC

[0089] Actions Action 1 Action 1 Action 2 Action 2 Action 3 Action 3 Action-wise Condition Condition Condition Condition Condition Condition Conditional 1a (Low 1 b (Low 2a 2b 3a 3b Clauses KPI1, KPI1,

[0090] (CCs) Medium KPI High KPI

[0091] 2, High KPI 2)

[0092] 3)

[0093] Cell # 001 60% 40% 33% 66% 20% 80%

[0094] Cell # 00X

[0095]

[0096] In step 413, node 701 appends conditional clauses for each cell to goal summary for complementing the goal summary with conditional clauses for each cell (identified using the identifier “Cell Id”). A combination of the goal summary and the conditional clauses for each cell results in the explanation information corresponding to goal-wise analysis or high-level summary or S1 summary. Explanation information corresponding to goal-wise analysis is made available to the user via the dashboard 400b in steps 414 and 415.

[0097] The goal-wise analysis depicted in Figure 4 can be performed periodically. The goalwise analysis depicted in Figure 4 can be performed periodically upon the user request. The user may request periodic goal-wise analysis in step 401.

[0098] Figure 5 depicts a process for generating explanation information corresponding to action-wise analysis or action-wise summary or level II summary (S2) according to the first and second embodiments of the invention. Level II summary is generated using the two modules, AGA 500d and ACC 500e. Node 701 executes modules AGA 500d and ACC 500e.

[0099] As shown in Figure 5, a user 500a may transmit, 501, to the node 701 via a dashboard 500b, the request indicating the type of analysis to be performed on the RL model configured to perform a task associated with one or more communication network cellsin the communication network 700. The dashboard 500b serves as an interface between the user 500a and the node 701. The user 500a may transmit, 501, the request to the dashboard 500b and from the dashboard 500b the request may be transmitted, 502, to the node 701. The node 701 obtains the request indicating the type of analysis to be performed on the RL model configured to perform a task associated with one or more communication network cells in the communication network 700. The type of analysis comprises action-wise analysis. For deeper analysis of the RL model, action-wise analysis is performed. Action-wise analysis is performed if a goal, associated with an action proposed by the RL model, was not fulfilled for a specific cell in the one or more communication network cells. The AGA 500d module obtains, 503, analysis information comprising the RL model and communication network data used for training the RL model. According to the first embodiment, the AGA500d module of the node 701 may comprise the analysis information. According to the second embodiment, the AGA 500d module of the node 701 may obtain the analysis information from a database 500c. The node 701 may be associated with the database 500c. The ACC 500e module obtains, 504, analysis information comprising the RL model, communication network data used for training the RL model, states associated with the RL model, a set of actions proposed by the RL model during RL model execution in the communication network environment and predicates associated with the RL model. According to the first embodiment, the ACC 500e module of the node 701 may comprise the analysis information. According to the second embodiment, the ACC 500e module of the node 701 may obtain the analysis information from the database 500c. The node 701 may be associated with the database 500c.

[0100] The node 701 performs the requested type of analysis, using the analysis information, with respect to a specific cell of the one or more communication network cells for each action in the set of actions proposed by the RL model. Performing the requested type of analysis comprises performing RD with respect to that specific cell of the one or more communication network cells for each action in the set of actions proposed by the RL model. Performing RD with respect to that specific cell for each action in the set of actions proposed by the RL model comprises identifying, 505, reward components associated with that specific cell for each action in the set of actions proposed by the RL model. Performing RD with respect to the specific cell comprisescomputing, 506, reward composition associated with that specific cell for each action in the set of actions proposed by the RL model. Performing RD with respect to that specific cell comprises generating, 507, goal summary for that specific cell for each action in the set of actions proposed by the RL model. RD with respect to that specific cell is performed by the AGA module 500d of node 701. Performing RD with respect to that specific cell of the communication network 700 enables the determination of whether a goal has been satisfied by that specific cell or not for each action in the set of actions proposed by the RL model. For an <action> from the set of actions, if it is determined that the goal is satisfied or met, the goal summary for that specific cell and for that <action> is transmitted, 508, to dashboard 500b made available to the user 500a via the dashboard 500b. For the <action> from the set of actions, if it is determined that the goal is not satisfied or not met, the goal summary for that specific cell and for that <action> is transmitted, 509, to the ACC 500e module to determine the conditional clauses.

[0101] The ACC module 500e performs APE with respect to that specific cell and for that <action>. Performing with respect to that specific cell and for that <action> comprises initiating, 510, Type 1 query “When does the RL agent propose the <action> from the set of actions?” for that specific cell. For that specific cell, the ACC module 500e of node 701 identifies, 511, target states for that <action>. For that specific cell, the ACC module 500e of node 701 generates, 512, conditional clauses for the <action>. Target states refers to a set of states, associated with the RL model, causing the RL model to propose the <action> during execution. Conditional clauses are generated by analysing the most frequent states in the target states causing the RL model to propose that <action> for that specific cell. In step 513, node 701 appends conditional clauses for that specific cell associated with the <action> to goal summary for complementing the goal summary with conditional clauses. A combination of the goal summary and the conditional clauses for that specific cell associated with the <action> results in the explanation information corresponding to action-wise analysis or actionwise summary or S2 summary. Explanation information corresponding to action-wise analysis is made available to the user via the dashboard 500b in steps 514 and 515. Outcome of action-wise analysis in the form of action-wise summary is presented in Table 3.The action-wise analysis depicted in Figure 5 can be performed periodically. The action-wise analysis depicted in Figure 5 can be performed periodically upon the user request. The user may request periodic action-wise analysis in step 501.

[0102] Table 3: Outcome of action-wise analysis (action-wise summary (S2))

[0103] Cell Id Date Action Goal Goal met Conditional

[0104] Clauses

[0105] Cell #... KPI1 High,

[0106] OOX KPI2 Low

[0107]

[0108] Figure 6 depicts a process for generating explanation information corresponding to state-wise analysis or state-wise analysis summary or level III summary (S3) according to the first and second embodiments of the invention. State-wise analysis is performed to obtain differences between observed and expected behaviour of the RL model using action-wise analysis and a State-wise Difference between observed and expected Actions, SDA, module 600d comprised in the node 701.

[0109] As shown in Figure 4, a user 600a may transmit, 601a, to the node 701 via a dashboard 600b, the request indicating the type of analysis to be performed on the RL model. The user 600a may transmit, 601a, the request to the dashboard 600b and from the dashboard 600b the request may be transmitted, 601a, to the node 701. The type of analysis comprises state-wise analysis. The SDA module 600d, performs analysis for each state belonging to a specific cell, using the information provided by S2 obtained by performing action-wise analysis. The SDA600d module obtains, 602, analysis information comprising the RL model, communication network data used for training the RL model states associated with the RL model, a set of actions proposed by the RL model during RL model execution in the communication network environment, predicates associated with the RL model, S2 summary and an expected action. According to the first embodiment, the SDA 600d module of the node 701 may comprise the analysis information. According to the second embodiment, the SDA 600d module of the node 701 may obtain the analysis information from a database 600c.The SDA module 600d initiates, 603, state-wise analysis. For each state associated with a communication network cell, the SDA module 600d determines if a designated goal is satisfied or not. If the goal is satisfied, the SDA module transmits, 604, the S1 or S2 summary to the user 600a via the dashboard 600b. If the goal is not satisfied, the SDAmodule 600d initiates, 605, the Type II query “Why didn’t the RL agent perform <expected action> in <state #>?”. By initiating the Type II query, the SDA module 600d identifies difference in states for the <expected action> and a <proposed action>. The SDA module 600d obtains an expected state vector associated with the <expected action>. The expected state vector corresponds to conditions of the communication network environment associated with a specific cell that would have led the RL agent to propose the <expected action>. The SDA module 600d obtains an observed state vector associated with the <proposed action>. The observed state vector corresponds to conditions of the communication network environment associated with the specific cell that caused the RL agent to propose the <proposed action>. In step 606, identifying the difference in states comprises the SDA module 600d identifying the difference between the observed state vector and the expected state vector. Using the identified differences in state vectors, the SDA module 600d in step 607 generates clauses for the <expected action>. Generating clauses comprises determining changes to the observed state vector that would have caused the RL agent to propose the <expected action>. The changes to the observed state vector are determined based on the identified differences between the observed state vector and the expected state vector. The generated clauses are appended, 608, to the S3 summary. The clauses are generated and appended for each state associated with a cell. Explanation information corresponding to state-wise analysis is made available to the user via the dashboard 600b in steps 609 and 610. Outcome of state-wise analysis in the form of state-wise summary is presented in Table 4.

[0110] The state-wise analysis depicted in Figure 6 can be performed periodically. The statewise analysis depicted in Figure 6 can be performed periodically upon the user request. The user may request periodic state-wise analysis in step 601.

[0111] The state-wise analysis can also be triggered every time when action-wise analysis is performed and S2 is generated. In such a scenario, S2 is appended with S3.Table 4: Outcome of the state-wise analysis (S3)

[0112] Cell ID State Action Expected Observed Expected Difference # ooX (Observed) Action state state

[0113] (KPIs) (KPIs)

[0114] State Uptilt Downtilt V = (V1, V = (VT, |V’ -V|

[0115] V2 V2',

[0116] # 00X1

[0117] V3,.. Vn) V3',...,

[0118] Vn')

[0119]

[0120] The goal-wise analysis depicted in Figure 4, the action-wise analysis depicted in Figure 5 and the state-wise analysis depicted in Figure 6 and their respective explanation information S1, S2 and S3 are used to updating the RL model. Upon performing the one or more analysis, the node 701 may trigger an automatic retraining of the RL model if there are unexpected outcomes in the one or more analysis. The three types of analysis and corresponding explanation information are either used individually or together in updating the RL model. For example, the goal-wise analysis can be used to select the subset of communication network cells where one or more goals were not satisfied, collect communication network data corresponding to those selected subset of cells and train the RL model with the collected communication network data corresponding to those selected subset of cells where one or more goals were not satisfied. In this manner, the node 701 may finetune the RL model with respect those cells where one or more goals were not satisfied.

[0121] The proposed types of analysis can be utilized to analyse the live-communication network more dynamically over a preferable period. The explanation information are presented in three different levels- goal-wise, action-wise and state-wise, thus, enabling a user to analysis or understand the decisions or actions proposed by the RL model in a more improved manner. The proposed solution enables a more improved process of troubleshooting in case of unexpected outcomes in one or more analysis. The explanation information obtained through the proposed solution are provided on a cell-level basis, thereby, enabling granular explanation information useful to identify problems in RL model execution on a cell-level basis. The proposed solution of XRL improves trust in customers when using RL models in the communication network.The explanation information are presented to the user in a tabular format, on a celllevel basis (granular) and with respect to state, actions and goals associated with the RL model. Thus, the proposed solution enables the user to better understand the working of the RL model in the communication network.

[0122] Figure 7 illustrates an example of the node 701 according to the embodiments of the present invention. The node 701 illustrated in Figure 7 implements the method 100 as illustrated in Figures 1, 4-6, for example on receipt of suitable instructions from a computer program 702. The node 701 comprises a processor or processing circuitry 703, and a computer program product 705 in the form of a memory 704. The processing circuitry 703 is operable to perform some or all of the steps of the method 100 as illustrated in Figures 1, 4-6, respectively. The memory 704 contains instructions executable by the processing circuitry 703 such that the node 701 is operable to perform some or all of the steps of the method 100 as illustrated in Figures 1, 4-6, respectively. The instructions may also include instructions for executing one or more telecommunications and / or data communications protocols. The instructions may be stored in the form of the computer program 702. In some examples, the processor or processing circuitry 703 may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, etc. The processor or processing circuitry 703 may be implemented by any type of integrated circuit, such as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), etc. The memory 704 may include one or several types of memory suitable for the processor, such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, solid state disk, hard disk drive, etc.

[0123] The communication network 700 illustrated in Figure 7 enables connectivity between user equipment, UE, and communication network nodes, such as the node 700. In that sense, the communication network 700 may be a 3rd Generation Partnership Project, 3GPP, network and may be configured to operate according to predefined rules or procedures, such as specific standards that comprise, but are not limited to: Global System for Mobile Communications, GSM; Universal Mobile Telecommunications System, UMTS; Evolved Packet System, EPS, 5th Generation System, 5GS, or any applicable future generation standard (e.g.,3GPP 6G). The communication network 700 here comprises one or more radio access network, RAN,nodes, and one or more core network nodes of an Evolved Packet Core, EPC, or an 5th Generation Core, 5GC. A RAN node may comprise a base station, eNodeB of an LTE network, gNodeB of a New Radio, NR, network, or any other current or future implementation of functionality facilitating the exchange of radio network signals between nodes of the communication network 700 and / or UE. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the communication network 700 comprises one or more Open-RAN, ORAN, network nodes. An ORAN network node is a node in the communication network 700 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the communication network, including one or more access network nodes and / or core network nodes.

[0124] The node 701 can a RAN node in the communication network 700. The node 701 can be a core network node in the communication network 700. The node 701 can be an Orchestration and Management, OAM, node. The node 701 can be a Service Management and Orchestration, SMO, node. The node 701 can be a non-real-time RAN Intelligent controller, Non-RT RIC.

[0125] The method 100 and its embodiments can be defined as a rApp on the Non-RT RIC within SMO framework. The SMO framework provides openness and offers an automation platform that is cloud native whereas, as per O-RAN principles, interfaces are available that enable connectivity and inter-operation between SMO, RAN functions and other applications. The Non-RT RIC offers the ability to gather data from various sources both from the radio and external sources as well as host rApps. The SMO framework supports interfaces towards O-RAN. An explainability rApp can be included as part of the rApp suite which could be accessed via other rApps. Other rApps can pass on ML based scores to the explainability rApp which can decipher these scores in a human understandable format. The explainability rApp can also publish this information via a UI or similar for user consumption. Another way to implement this explainability solution is within the platform services of non RT-RIC.This could be coupled with the offered ML model within the platform services. This explainability platform service can be invoked by any rApp similar to how Al model platform services are invoked. Explainability framework, proposed in this invention, likely to be a part of 4G and 5G cell shaping rApp in any optimization non-Real time RIC. Extension of the existing solutions for network optimization to explainability through a periodically updatable dashboard, running online. This would automate the existing manual process of goal analysis by radio domain experts. Facilitating multilevel, consolidated, granular explanation reports, applicable to live network at scale for gaining deeper insights in the process of network optimization using RL / ML. This would facilitate to generate explanation information for a specific set of states, specific to a particular cell in the communication network.

[0126] It is to be understood that the singular form "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It will be further understood that the terms, "comprises" "comprising", "includes" and / or "including" when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. REFERENCES

[0127] 1. B. Hayes and J. A. Shah, “Improving robot controller transparency using autonomous policy explanations”, 12th ACM / IEEE International Conference on Human-Robot Interaction (HRI), IEEE, 2017, pp. 303-312. 2017.

[0128] 2. Z. Juozapaitis, A. Koul, A. Fem, M. Erwig, and F. Doshi-Velez, “Explainable reinforcement learning via reward decomposition,” in IJCAI / ECAI Workshop on explainable artificial intelligence, 2019.

Claims

Claims1. A computer-implemented method (100) for performing explainable reinforcement learning, RL, XRL, in a communication network (700), the method (100) comprising:- obtaining (101) a request indicating a type of analysis to be performed on an RL model configured to perform a task associated with one or more communication network cells in the communication network (700);- performing (103) the requested type of analysis, using analysis information, with respect to at least one cell of the one or more communication network cells;- compiling (104) an explanation information based on analysis results obtained by performing the requested type of analysis with respect to the at least one cell of the one or more communication network cells;- updating (105) the RL model based on the compiled explanation information.

2. The method (100) according to any of the preceding claims, wherein the type of analysis is one of a goal-wise analysis, state-wise analysis and action-wise analysis.

3. The method (100) according to any of claims 1-2, wherein the analysis information comprises one or more of:- the RL model,- communication network data used in RL model training,- a set of actions proposed by the RL model,- states associated with the RL model,- predicates associated with the RL model4. The method (100) according to claim 3, comprising:- obtaining (102) the analysis information.

5. The method (100) according to any of the preceding claims, wherein performing (103) the requested type of analysis comprises performing therequested type of analysis with respect to each cell of the one or more communication network cells.

6. The method (100) according to any of the preceding claims, wherein compiling (104) the explanation summary comprises compiling the explanation summary based on analysis results obtained by performing the requested type of analysis with respect to each cell of the one or more communication network cells.

7. The method (100) according to any of the preceding claims, wherein the performing (103) the requested type of analysis comprises performing reward decomposition, RD.

8. The method (100) according to any of the preceding claims, wherein the performing (103) the requested type of analysis comprises performing autonomous policy explanation, APE.

9. The method (100) according to any of the preceding claims, comprising:- determining whether one or more goals of the task have been satisfied by the RL model.

10. The method (100) according to any of the preceding claims, wherein the obtained request indicates two or more types of analysis to be performed.

11. The method (100) according to any of the preceding claims, comprising:- performing the requested type of analysis periodically.

12. The method (100) according to any of the preceding claims, wherein the task comprises radio network optimization.

13. The method (100) according to any of the preceding claims, wherein the task comprises machine learning safety shield, MLSS, based radio network optimization.

14. The method (100) according to any of the preceding claims, wherein the RL model is a trained RL model.

15. The method (100) according to any of the preceding claims, wherein the method is defined as a rApp on a non-real time radio access network, RAN, intelligent controller, non-RT RIC, within a service management and orchestration system framework.

16. The method (100) according to any of the preceding claims, wherein the method (100) is performed by a node (701) in the communication network (700).

17. The method (100) according to claim 15, wherein the node (701) is one of:a RAN node,a core network node,an Orchestration and Management, OAM, node, ora Service Management and Orchestration, SMO, node.

18. A node (701) for performing explainable reinforcement learning, RL, XRL, in a communication network (700), the node (701) comprising processing circuitry (703) and a memory (704), the memory (704) containing instructions executable by the processing circuitry (703) such that the node (701) is operable to:- obtain (101) a request indicating a type of analysis to be performed on an RL model configured to perform a task associated with one or more communication network cells in the communication network;- perform (103) the requested type of analysis with respect to at least one cell of the one or more communication network cells;- compile (104) an explanation summary based on analysis results obtained by performing the requested type of analysis with respect to the at least one cell of the one or more communication network cells;- update (105) the RL model based on the explanation summary.

19. The node (701 ) of claim 17, further operable to perform a method (100) according to any of claims 2-16.

20. The node (701) according to any of claims 18-19, wherein the node (701) is one of:a radio access network, RAN, node,a core network node,an Orchestration and Management, OAM, node, ora Service Management and Orchestration, SMO, node.

21. A computer program (702), comprising instructions which when run on a processor (703) of a node (701), causes the node (701) to perform a method (100) according to any of claims 1 -17.

22. A computer program product (705) which comprises a computer readable storage medium (704) on which a computer program (702) according to claim 21 is stored.

Images