Method and network entity for evaluating trustworthiness of a network digital twin of a communication network

Abstract

Disclosed is a method performed by one or more network entities (160; 170) for evaluating trustworthiness of a network digital twin (150) set up as a digital twin of a part (130) of a communication network (100) The method comprises determining an expected entity future state of the part (130) of the communication network (100) based on data associated with a current state and on data associated with historical states of the part (130) of the communication network and receiving an expected digital twin future state of the part (130) of the communication network (100) simulated by the network digital twin (150). The method further comprises determining a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, and informing a concerned party (140; 180) of the determined trustworthiness score.

Description

METHOD AND NETWORK ENTITY FOR EVALUATING TRUSTWORTHINESS OF A NETWORK DIGITAL TWIN OF A COMMUNICATION NETWORKTECHNICAL FIELD

[0001] The present disclosure relates generally to methods and network entities for evaluating trustworthiness of a network digital twin set up as a digital twin of at least a part of a communication network. The present disclosure further relates to computer programs and carriers corresponding to the above methods and network entities.BACKGROUND

[0002] The concept of Digital Twin (DT) has attracted many business areas since its revolutionary way of maintenance of a physical system. A Digital Twin is defined in literature as a virtual synchronous replica model that clones a real object, a process, a service, or other abstraction. A DT updates this virtual replica model with any new configurations, events, or changes seen in the actual world through two-way communication between the virtual replica model and real-world assets. A Network Digital Twin (NDT), is the result of DTs' expansion into a variety of domains, including networks. The aim of the NDT concept is to achieve closed-loop network administration easier. For this purpose, NDTs are suggested and used to facilitate different use case scenarios such as intent-based networking, innovative network security tactics, and effective network management and administration.

[0003] NDTs are expected to be widely deployed in next generation communication networks, such as fifth generation (5G) and sixth generation (6G) wireless communication networks. NDT development fundamentally involves several steps including data collection, data validation, knowledge extraction, model development. When an NDT is implemented, it is assumed that NDT is correctly operating as a synchronous digital replica of the physical network where changes in the physical network are reflected into it. At this point, being sure about (i) whether the developed NDT correctly reflects the physical network or not, and (ii) is not compromised, are important concerns for the NDT’s trustworthiness andreliability. In “Validation of Digital Twins; Challenges and opportunities”, by Hua, E. Y. et al, published in “Proceedings of the 2022 Winter Simulation Conference, WSC 2022”, pages 2900-2911 , published in 2023 by IEEE, a validation process is proposed to ensure an NDT model’s reliability, robustness, and trustworthiness. In the described validation process, the NDT model is executed with an initial set of already known input-output dataset of the physical network to ensure the NDT accurately reflects the physical network. After the initial validation of the NDT model, a dynamic validation process is conducted to align the physical network with its digital replica. This dynamic validation process utilizes (i) continuous data collection from the physical network, (ii) extraction of the model and its components such as model’s structure and topology in the form of probabilities, constants, and probability distributions. The validation process also uses expert-in-the-loop for the relevant events and data streams specification.

[0004] The accuracy of this validation process involves evaluating how well the results of a digital twin align with real-world data to ensure that the simulation accurately represents the actual behavior of the network. However, this approach requires ongoing validation with real-world data, which can be time-consuming and computationally expensive. Further, even though the model is validated accurately and correctly this validation process still captures one snapshot of the physical system, which does not result as a validation of many or all states, time, possible scenarios or rare events that may occur. Further, this validation process relies on a real-time data stream from the physical environment and assumes this is always reliable so that accurate data is provided. However, this might not be true. Also, the validation process requires final human intervention. So, an improved way of evaluating trustworthiness of a network digital twin is needed.SUMMARY

[0005] It is an object of embodiments of the invention to address at least some of the problems and issues outlined above. It is an object of embodiments of the invention to provide a way of evaluating trustworthiness of a network digital twin(NDT) that provides real-time robustness, that is cost-effective, that brings lower computational load than prior art, that has a high adaptiveness and / or that is scalable. It is possible to achieve at least of one of these objects by using methods, and network entities as defined in the attached independent claims.

[0006] According to one aspect, a method is provided that is performed by one or more network entities for evaluating trustworthiness of a network digital twin set up as a digital twin of at least a part of a communication network. The method comprises obtaining, from the at least part of the communication network, current data associated with a current state of the at least part of the communication network, and determining an expected entity future state of the at least part of the communication network at a future time point, based on the obtained current data associated with the current state and on historical data associated with one or more historical states of the at least part of the communication network obtained at one or more time points during a time period. The method further comprises receiving, from the network digital twin, an expected digital twin future state of the at least part of the communication network at the future time point, wherein the expected digital twin future state is simulated by the network digital twin based on the current data associated with the current state that the network digital twin has received from the at least part of the communication network, determining a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, and selectively informing a concerned party of the determined trustworthiness score.

[0007] According to another aspect, one or more network entities is provided that is configured for evaluating trustworthiness of a network digital twin set up as a digital twin of at least a part of a communication network. The one or more network entities comprises a processing circuitry and a memory. Said memory contains instructions executable by said processing circuitry, whereby the one or more network entities is operative for obtaining, from the at least part of the communication network, current data associated with a current state of the at least part of the communication network, and determining an expected entity future state of the at least part of the communication network at a future time point,based on the obtained current data associated with the current state and on historical data associated with one or more historical states of the at least part of the communication network obtained at one or more time points during a time period. The one or more network entities is further operative for receiving, from the network digital twin, an expected digital twin future state of the at least part of the communication network at the future time point, wherein the expected digital twin future state is simulated by the network digital twin based on the current data associated with the current state that the network digital twin has received from the at least part of the communication network. The one or more network entities is further operative for determining a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, and selectively informing a concerned party of the determined trustworthiness score.

[0008] According to other aspects, computer programs and carriers are also provided, the details of which will be described in the claims and the detailed description.

[0009] Further possible features and benefits of this solution will become apparent from the detailed description below.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The solution will now be described in more detail by means of exemplary embodiments and with reference to the accompanying drawings, in which:

[0011] Fig. 1 is a schematic diagram of a communication network comprising a trust evaluator according to the present invention.

[0012] Fig. 2 is a flow chart illustrating a method performed by one or more network entities, according to possible embodiments.

[0013] Fig. 3 is a schematic block diagram of entities involved in a method according to embodiments, including signals sent between the entities.

[0014] Fig. 4 is a signaling diagram illustrating an example of a procedure according to further possible embodiments.

[0015] Fig. 5 is another flow chart illustrating a method according to embodiments.

[0016] Fig. 6 is a block diagram illustrating one or more network entities in more detail, according to further possible embodiments.DETAILED DESCRIPTION

[0017] Fig. 1 shows a communication network 100 comprising a radio access network (RAN) 130 comprising one or more RAN nodes aka network nodes 132 that is in, or is adapted for, wireless communication with a wireless communication device aka wireless device 140. The network node 132 provides radio access in a cell 145 covering a geographical area. The communication network 100 of fig. 1 further comprises a network digital twin (NDT) 150 that operates as a synchronous digital replica of at least a part of the communication network 100, for example a replica of the RAN 130. The NDT 150 is here a part of the communication network 100, however, the NDT 150 might as well be situated outside the communication network 100. According to embodiments, and as will be further described below, the communication network 100 further comprises or is connected to a trust evaluator 160, 170. The trust evaluator 160, 170 is connected to the part of the network for which the NDT 150 operates as a digital twin, for example the RAN 130. The trust evaluator 160, 170 is further connected to the NDT 150. However, the trust evaluator 160, 170 is preferably kept separate from the NDT 150 to work independently from the NDT 150. The communication network 100 further comprises, or is connected to, a cloud network 175. The trust evaluator may be situated in the cloud network 175. Such trust evaluator has got reference number 170 in fig. 1. Alternatively, the trust evaluator is a separate node in or connected to the communication network 100. Such trust evaluator has got reference number 160 in fig. 1. The communication network 100 may further comprise an operationand maintenance node or function 180. The trust evaluator is also called “one or more network entities” in the disclosure.

[0018] The communication network 100 may be any kind of communication network that can provide radio access to wireless devices. Example of such communication networks are networks based on Global System for Mobile communication (GSM), Enhanced Data Rates for GSM Evolution (EDGE), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA 2000), Long Term Evolution (LTE), LTE Advanced, Wireless Local Area Networks (WLAN), Worldwide Interoperability for Microwave Access (WiMAX), WiMAX Advanced, as well as fifth generation (5G) communication networks based on technology such as New Radio (NR), and any possible future sixth generation (6G) communication network.

[0019] The network node 132 may be any kind of network node that can provide wireless access to a wireless device 140 alone or in combination with another network node. Examples of network nodes 132 are a base station (BS), a radio BS, a base transceiver station, a BS controller, a network controller, a Node B (NB), an evolved Node B (eNB), a gNodeB (gNB), a Multi-cell / multicast Coordination Entity, a relay node, an access point (AP), a radio AP, a remote radio unit (RRU), a remote radio head (RRH), a multi-standard BS (MSR BS), a WLAN Access Point (AP), a router, a Bluetooth AP, Open RAN (ORAN) network nodes such as ORAN Radio Unit (O-RU), ORAN Distributed Unit (O-DU), ORAN eNB (O-eNB), etc.

[0020] The wireless device 140 may be any type of device capable of wirelessly communicating with a network node 130 using radio signals. For example, the wireless device 140 may be a User Equipment (UE), a machine type UE or a UE capable of machine to machine (M2M) communication, a sensor, a tablet, a mobile terminal, a smart phone, a laptop embedded equipped (LEE), a laptop mounted equipment (LME), a USB dongle, a Customer Premises Equipment (CPE), an Internet of Things (loT) device, etc.

[0021] Fig. 2, in conjunction with fig. 1 , illustrates a method performed by one or more network entities 160; 170 for evaluating trustworthiness of a network digital twin 150 set up as a digital twin of at least a part 130 of a communication network 100. The method comprises obtaining 204, from the at least part 130 of the communication network 100, current data associated with a current state of the at least part 130 of the communication network 100, and determining 206 an expected entity future state of the at least part 130 of the communication network 100 at a future time point, based on the obtained current data associated with the current state and on historical data associated with one or more historical states of the at least part 130 of the communication network 100 obtained at one or more time points during a time period. The method further comprises receiving 208, from the network digital twin 150, an expected digital twin future state of the at least part 130 of the communication network 100 at the future time point, wherein the expected digital twin future state is simulated by the network digital twin 150 based on the current data associated with the current state that the network digital twin 150 has received from the at least part 130 of the communication network 100, determining 210 a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, and selectively informing 212 a concerned party 140; 180 of the determined trustworthiness score 210.

[0022] The at least part of the communication network for which the digital twin is set-up could be for example the RAN 130 of the communication network 100 or a part of the RAN 100 but also a core network (CN) of the communication network 100 or any other part of the communication network, or even the whole communication network 100. The current data and / or the historical data can be obtained from one or more different parts of the network such as the CN, the RAN, user plane and / or control plane. The obtained current data and / or the historical data may be associated with a configuration of the at least part of the network. The current data and / or the historical data could be one or more of measurement data, functional data, operational data, topological data. The current data and / or the historical data could be "contextual data" that provides insights into external factors like user behavior, service data etc. It may also be "use case specific data",like for monitoring and predicting for example throughput of the network. So, then throughput-related data is collected and processed. The expected entity future state of the at least part 130 of the communication network 100 determined by the one or more network entities 160; 170 may be for example throughput, latency and / or Quality of Service (QoS) provided by the at least part of the communication network. The expected digital twin future state received from the network digital twin 150 may be for example throughput, latency and / or Quality of Service (QoS) provided by the at least part of the communication network.

[0023] The historical data associated with one or more historical states of the at least part 130 of the communication network 100, such as historical measurement data and historical operational data, may be stored in a database communicatively connected to the one or more network entities 160; 170. Generally speaking, the more time points for which historical data is obtained, the better the determination of the expected future state i.e. the more historical states for which historical data has been obtained, the better the determination of the expected future state. The time period during which historical data is obtained may vary between microseconds to days. The time period may depend on the load of the part of the communication network and / or the type of the activity. The trustworthiness score may be a measure on how well the expected future state determined by the network digital twin corresponds to the expected future state determined by the one or more network entities. E.g., the trustworthiness score is determined by comparing the expected future state determined by the network digital twin with the expected future state determined by the one or more network entities, i.e. the trust evaluator. Then the same characteristics determined by the network digital twin and the trust evaluator is / are compared, i.e., latency determined by the network digital twin 150 is compared to latency determined by the trust evaluator 160; 170, and / or throughput determined by the network digital twin 150 is compared to throughput determined by the trust evaluator, and / or QoS determined by the network digital twin 150 is compared to QoS determined by the trust evaluator.

[0024] Selectively informing 212 a concerned party 140; 180 of the determined trustworthiness score 210 may signify that the trust evaluator has instructions to inform the concerned party at certain occasions, such as when the trustworthiness score is lower than a certain threshold indicating a possible issue with the NDT, i.e. that the difference between the throughput / latency / QoS determined by the NDT differs too much from the throughput / latency / QoS determined by the trust evaluator. Alternatively, the selectively informing of the trustworthiness score may signify informing the concerned party at certain defined time points, or regularly, independent of any threshold. The concerned party 140; 180 may be related to the owner of the communication network or may be any other stakeholder or customer. The concerned party may be informed by sending a message containing the trustworthiness score to a communication device 140, e.g., computer, mobile phone, etc., of the party or to a communication node of the communication network, e.g. the O&M node 180 of fig. 1.

[0025] Such a method provides real-time robustness by giving concerned parties, e.g., users of the NDT, insights to the trustworthiness of the NDT. Further, such a method does not need to calibrate parameters of the NDT consistently, as in prior art, which brings down computational load. Also, the method is applicable for various NDTs not only for specific NDT architectures.

[0026] According to an embodiment, the one or more network entities 160; 170 comprises a belief network structure. Further, the expected entity future state is determined 206 by the one or more network entities using the belief network structure. According to an embodiment, the one or more network entities 160; 170, i.e., the trust evaluator, is constructed on top of a belief network structure. A belief network structure is widely used in predictive analysis for complex / large systems. A belief network, aka Bayesian network, is in Wikipedia described as “a probabilistic graphical model that represents a set of variables andtheir conditional dependencies via a directed acyclic graph (DAG). While it is one of several forms of causal notation, causal networks are special cases of Bayesian networks. Bayesian networks are ideal for taking an event that occurred and predicting the likelihood that any one of several possible known causes was thecontributing factor.” The one or more network entities may use the belief network structure for determining the expected entity future state by feeding the belief network structure with the historical state data and the current state data and the belief network structure provides the determ ined / expected future state as a result.

[0027] According to an embodiment, the selectively informing 212 a concerned party 140; 180 of the determined trustworthiness score comprises only informing the concerned party 140; 180 when the determined trustworthiness score indicates a trustworthiness of the expected digital twin future state lower than a defined threshold. By only informing the concerned party when there seems to be low trustworthiness, such as when e.g. the latency determined by the trust evaluator 160, 170 differs more than a defined threshold from the latency determined by the NDT, amount of data to be set over the network can be limited.

[0028] According to another embodiment, the one or more network entities 160; 170 is situated in one or more network functions of a core network of the communication network 100 or in a service management and orchestration (SMO) network function 1 of the RAN 130 of the communication network 100. The core network (CN) and the SMO are trustful environments in the communication network 100. They are as such not related to the NDT and to the application layer. Hereby, security can be on a high level, as any security breaches in the application layer or in the NDT itself cannot be spread to the one or more entities.

[0029] According to another embodiment, the expected entity future state and the expected digital twin future state each comprises a plurality of state characteristics, wherein the trustworthiness score is determined 210 by comparing one or more state characteristics of the plurality of state characteristics of the expected entity future state with same one or more state characteristics of the plurality of state characteristics of the expected digital twin future state. The state characteristics may be one or more of e.g. latency, throughput, QoS. The comparing of state characteristics is then performed by comparing latency determined by the NDT with latency determined by the trust evaluator, throughput determined by the NDT with throughput determined by the trust evaluator and / or QoS determined by the NDT with QoS determined by the trust evaluator.

[0030] According to an alternative of this embodiment, the comparing of the one or more state characteristics of the plurality of state characteristics of the expected entity future state with same one or more state characteristics of the plurality of state characteristics of the expected digital twin future state comprises: calculating a mean absolute error (MAE), of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state; calculating a coefficient of determination (R2) of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state, and calculating the trustworthiness score based on the calculated MAE and the calculated R2.

[0031] According to another embodiment, the method further comprises plotting the calculated MAE and the calculated R2as a histogram. Further, the calculating of the trustworthiness score based on the calculated MAE and the calculated R2is performed by analyzing the histogram.

[0032] According to yet another embodiment, the determining 206 of an expected entity future state comprises determining a plurality of expected entity future states for each of a plurality of scenarios and wherein the receiving 208 of an expected digital twin future state comprises receiving a plurality of expected digital twin future states for each of the plurality of scenarios, and wherein the trustworthiness score is determined 210 based on the plurality of expected entity future states and the plurality of expected digital twin future states. Hereby, it can be tested how well the NDT coincides with the trust evaluator for different scenarios.

[0033] According to yet another embodiment shown in fig. 2, the one or more network entities 160; 170 further comprises, or is communicatively connected to, a database. Further, the method comprises obtaining 201 the historical data associated with historical states of the at least part 130 of the communication network 100 at the one or more time points during a time period, before the obtaining 204 of the current data associated with the current state, and storing 202 the obtained historical data in the database.

[0034] According to yet another embodiment, which also is shown in fig. 2, the method further comprises informing 214 the concerned party of the determined 206 expected entity future state at the future time point, and receiving 216, after or at the future time point, an indication associated with a correctness of the determined expected entity future state at the future time point in relation to an observed actual state at the future time point. The indication associated with correctness can be provided by a system determining the actual state or it may be provided by a user, such as the concerned party. By such feedback, the one or more network entities can adjust its determination of expected entity future state based on the received feedback and provide better determinations over time the more feedback it receives.

[0035] In the following, different embodiments of the invention are described. As mentioned, embodiments aim to create a trustworthiness score aka metric aka value to evaluate reliability of an NDT set up as a digital twin of at least a part of a communication network. To evaluate reliability or trustworthiness of the NDT, a separate trust evaluator block is proposed. The trust evaluator is preferably kept separate and independent from the NDT since the NDT might be compromised or does not reflect real network properly. According to embodiments, four nodes or layers are involved: (i) the physical infrastructure of the network, i.e. the at least part of the communication network; (ii) the NDT; (Hi) the trust evaluator, and (iv) an NDT application-access layer. The trust evaluator is in one embodiment constructed as a belief network structure, such as a Bayesian networks, that operates using the historical data and current observation measurements from the physical infrastructure. Belief network structures are already widely used in other areas for predictive analysis for complex physical environments and processes. It also used where the available data is limited or not qualified enough. We have discovered that belief network structures could be of great advantage for evaluating trustworthiness of an NDT. According to embodiments, while the NDT gets the data from the physical infrastructure and execute required simulations for the specific network configurations or to predict network’s future states, separate predictive analyses are conducted in the trust evaluator. The trust evaluator then compares the results from the NDT with its own results and provides atrustworthiness score of the NDT to a concerned stakeholder. The trustworthiness score may be compared against a predefined threshold; if the metric is below the threshold, the trust evaluator triggers a warning to the concerned stakeholder, signaling potential issues or risks associated with the NDT.

[0036] Fig. 3 shows an overall architecture in which embodiments of the present invention may be used. The overall architecture comprises physical network infrastructure to be evaluated, here exemplified by a RAN 302, as well as an NDT 304, a Trust evaluator 306 and an NDT application 308 aka NDT applicationaccess layer device.

[0037] The NDT 304 is in this example implemented and deployed for working as a digital twin of the RAN 302 of a communication network, however, the NDT 304 may work as a digital twin for any part of a communication network. In the context of at least some embodiments, the NDT 304 operates as usual. The NDT 304 is integrated with the RAN’s orchestrating and management lifecycle. In here, the NDT 304 receives 1.1 related data from RAN 302 to keep itself as a real-time digital shadow of the RAN. Therefore, there is no approximation or estimation process in the NDT 304 since it receives the actual required data from the physical part of the network. During the operating lifecycle, the NDT uses Artificial Intelligence / Machine Learning (AI / ML) methods to gain intuition about the condition of the RAN 302 and enable automation of the RAN management process. In this context, the main functionality of the NDT 304 here is executing the specific network simulations and give insights as well as expected states of the RAN 302 for the future (timepoint t+1 ) before deploying or implementing them in the actual RAN 302. This insights and simulation results, e.g. simulated states for timepoint t+1 are shared 1.2 with the related NDT application 308 that the NDT serves. Also, the simulation results are sent 1.2 to the RAN 302 to inform the actual network and to enable physical network evaluating the results of the NDT 304.

[0038] Predictive analyses are conducted inside the trust evaluator 306. The trust evaluator 306 is located in a secured, trustful environment such as a network function in a core network (CN) of the communication network, or inside a servicemanagement and orchestration (SMO), control unit (CU) in the RAN. The reason behind such a location is to mitigate third party attacks, since the NDT 304 is not only owned by the RAN 302 or the CN but also by the NDT user or owner, such as network operators, network administrators, and orchestrators. Therefore, the NDT 304 has an application layer interface (NDT Application 308) and interaction with other parties. If the trust evaluator 306 would have been located in the application layer level as well, then vulnerabilities in the application layer also would affect the trust evaluator 306. The trust evaluator 306 receives 1.3 data about the current state of the RAN (i.e. at time t), such as measurements from the RAN 302 as observation data of the current state of the RAN. The trust evaluator 306 performs 1.5 a predictive analysis. Some data aggregation and processing 1.4 of the received measurements may be needed to be performed by the trust evaluator 306 in connection with the predictive analysis. The trust evaluator 306 involves a belief network structure in the predictive analysis 1.6. As input to the predictive analysis 1.6, except for the data of the current state of the RAN, historical data 1.5 of historical states of the RAN are used, such as operational, maintenance, functional and / or failure data of historical states. The historical data may be stored in a database 307, which in this example resides in the trust evaluator 306 but which may be a separate entity. In the predictive analysis 1.6, an expected future state (at time +1) of the RAN is calculated and estimated within a probability through the belief network, based on the historical data, and the actual measurements from the RAN. Further, the trust evaluator 306 receives 1.2 from the NDT 304, the simulation results for the future timepoint t+1 executed in NDT including the simulated future states of the RAN.

[0039] Thereafter, the trust evaluator 306 compares 1.7 its predictive analysis results for time t+1 with the simulation results of the NDT 304 for t+1 and / or for specific network configurations. As mentioned, the NDT 304 is assumed untrustworthy in this structure. The trustworthiness score 1.8 is created by comparing the predictive analysis of the belief network and the simulation results received from the NDT. It can be initialized as zero and can be increased with alignments. Embodiments of the creation of the trustworthiness metric will be described further below.

[0040] After the trust evaluator 306 has created the trustworthiness score 1.8, the trust evaluator 306 informs 1.9 any stakeholder, i.e. a concerned party, such as a related NDT user at the application layer 308 about the trustworthiness score, i.e. as a trustworthiness score. This score may be provided with a safety interval. According to an embodiment, the trust evaluator alerts the NDT user if the value decreases below a predetermined threshold.

[0041] According to an embodiment, the proposed solution might be improved in an adaptive way. The related stakeholder located in the NDT’s application-access layer 308 might inform the trust evaluator 306 considering the actual states of the physical network infrastructure at t+1 since the user is able to observe the actual state of the network at the future timepoint when this timepoint occurs. The user could provide feedback 1.10 of the actual state at t+1 to the trust evaluator, and the trust evaluator 306 could use such feedback to improve its predictive analysis and / or adjust its trustworthiness score. Hereby, even in the existence of false positive cases, which are unavoidable in any solution, the system still evolve itself to reduce the occasions of them. According to another embodiment, the feedback could be as rewards or punishments to the trust evaluator 306. Further, as the RAN 302 continues performing measurements after time t, the measurements performed by the RAN 302 at time t+1 can be sent 1.11 to the trust evaluator 306 and used for adjusting the trustworthiness score for the coming estimations of trustworthiness.

[0042] Fig. 4 is a signaling diagram of signals sent between the entities of fig. 3, that is a RAN 401 , an NDT 402, a Trust evaluator 403 and an NDT application 404, according to embodiments. The sequence starts with the RAN 401 sending 2.1 data needed for the NDT 402 to perform its simulations, such as configuration data, real-time data, e.g. measurement data, service-related data and functional data. The RAN 401 further sends 2.2 measurements of the state of the RAN at time (t), i.e., measurements at time t. The trust evaluator 403 then performs its predictive analysis and determines 2.3 one or more expected states for time t+1. The NDT 402 performs 2.4 its simulations for t+1 based on the received data, maybe simultaneously as the trust evaluator 403 performs its predictive analysis.The NDT 402 sends 2.5 its simulation results for t+1 to the RAN 401 that can evaluate them. The NDT 402 also sends 2.6 the simulation results for t+1 to the trust evaluator 403. The trust evaluator 403 then compares 2.7 the simulation results for t+1 from the NDT, i.e. the expected digital twin future state with the determined expected state for t+1 that the trust evaluator itself has determined, i.e. the expected entity future state. The comparison results in a trustworthiness score or value for the NDT, or actually the digital twin future state. The Trust evaluator 403 may then alert or inform 2.8 a concerned party having an NDT application 404, such as a party of the RAN owner of the trustworthiness score or value. In an embodiment, the trust evaluator 403 only informs or alerts the concerned party when the trustworthiness score indicates doubtful trustworthiness of the NDT. There may also be a step of the RAN 401 sending 2.9 actual state data for time t+1 when the future time t+1 has occurred, which data can be used for evaluating the NDT and / or for evaluating the trustworthiness score determination.

[0043] In the following, an embodiment for determining or creating the trustworthiness is described, with reference to fig. 5. As mentioned, the trustworthiness score is created by comparing the predictive analysis of the trust evaluator, determined using e.g., a belief network, with the simulation results received from the NDT. This may be performed according to the following: A network scenario that the NDT performs and simulates is selected 502. This scenario is preferably one of the NDT’s tasks, such as throughput predictions, latency estimations or resource utilization metrics. Thereafter, the prediction(s) of the Trust Evaluator PTE is / are paired and compared 506 with the simulation result(s) of the NDT PNDTQSPNDT’ PTE')Then the mean absolute error (MAE) 508 is computed and an r-squared (R ) is computed for correlation analysis. The MAE for the paired PTE and PNDT is computed according to the following:wherein n is the total number of predictions, PTE( ) 'STrust Evaluator’s prediction results and P^DT(. 'SNDT’S simulation results. If n = 3, the predictions may be throughput, latency and resource utilization, and both the trust evaluator and the NDT has predicted or simulated all these three so that they can be paired.The R may be computed for correlation analysis according to the following:The MAE and the R are performed to have both an instantaneous and an overall evaluation. An instantaneous evaluation is the absolute error at a specific time step. This shows how well the digital twin's prediction matches reality at that moment. An overall evaluation is the average across all time steps. This provides an overall sense of how accurate the digital twin is. Error distributions can be plotted as a histogram and consistency can be analyzed. Also, this analysis can be conducted with a predefined confidence interval overlap depending on the scenario.

[0044] Further, such results may be obtained for more than one scenario. In such a case, the results for the more than one scenario can then be aggregated 510 and then divided 512 by the total number of scenarios. Then a trustworthiness score is obtained 514.

[0045] According to embodiments, a trustworthiness threshold is applied. If the trustworthiness score decreases below the trustworthiness threshold, the trust evaluator may alert the NDT application of the concerned user. This trustworthiness threshold can be initiated from zero since the solution is adaptive. The trust evaluator can adjust the threshold properly with feedback from its user, i.e. NDT application. Therefore, the trustworthiness score and the threshold could be initialized from zero, or be initialized randomly or be initialized from predetermined values, because they are adaptively updated with respect to the feedback as well as the comparison result. Shortly, if the results are aligned withinthe safety interval, then the trustworthiness score is updated and increased. In the reverse situation, the metric is updated again and decreased. These processes might be conducted automatically while the trustworthiness score stays above the threshold.

[0046] Fig. 6, in conjunction with fig. 1 , shows one or more network entities 160; 170 configured for evaluating trustworthiness of a network digital twin 150 set up as a digital twin of at least a part 130 of a communication network 100. The one or more network entities 160; 170 comprises a processing circuitry 603 and a memory 604. Said memory contains instructions executable by said processing circuitry, whereby the one or more network entities 160; 170 is operative for obtaining, from the at least part 130 of the communication network, 100 current data associated with a current state of the at least part 130 of the communication network 100, and determining an expected entity future state of the at least part 130 of the communication network 100 at a future time point, based on the obtained current data associated with the current state and on historical data associated with one or more historical states of the at least part 130 of the communication network 100 obtained at one or more time points during a time period. The one or more network entities 160; 170 is further operative for receiving, from the network digital twin 150, an expected digital twin future state of the at least part 130 of the communication network 100 at the future time point, wherein the expected digital twin future state is simulated by the network digital twin 150 based on the current data associated with the current state that the network digital twin 150 has received from the at least part 130 of the communication network 100. The one or more network entities 160; 170 is further operative for determining a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, and selectively informing a concerned party 140; 180 of the determined trustworthiness score.

[0047] The one or more network entities 160; 170 may be realized as a separate node 160 in the communication network 100 or in an existing node of the communication network 100. Alternatively, the one or more network entities 160,170 may be realized as a group of network nodes 170, wherein functionality of the one or more network entities 170 is spread out over the group of network nodes. The group of network nodes may be different physical, or virtual, nodes of the network. The group of network nodes may be situated in or belong to a cloud network 175.

[0048] According to an embodiment, the one or more network entities 160; 170 comprises a belief network structure. Further, the one or more network entities 160; 170 is operative to determine the expected entity future state using the belief network structure.

[0049] According to another embodiment, the one or more network entities 160; 170 is operative to perform the selectively informing of a concerned party 140; 180 of the determined trustworthiness score by only informing the concerned party 140; 180 when the determined trustworthiness score indicates a trustworthiness of the expected digital twin future state lower than a defined threshold.

[0050] According to another embodiment, the one or more network entities 160; 170 is situated in one or more network functions of a core network of the communication network 100 or in a service management and orchestration (SMO) network function of a RAN 130 of the communication network 100.

[0051] According to another embodiment, the expected entity future state and the expected digital twin future state each comprises a plurality of state characteristics. Further, the one or more network entities 160; 170 is operative to determine the trustworthiness score by comparing one or more state characteristics of the plurality of state characteristics of the expected entity future state with same one or more state characteristics of the plurality of state characteristics of the expected digital twin future state.

[0052] According to another embodiment, the one or more network entities 160; 170 is operative to compare the one or more state characteristics of the plurality of state characteristics of the expected entity future state with the same one or more state characteristics of the plurality of state characteristics of the expected digitaltwin future state by: calculating a mean absolute error (MAE) of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state; calculating a coefficient of determination (R2) of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state, and calculating the trustworthiness score based on the calculated MAE and the calculated R2.

[0053] According to yet another embodiment, the one or more network entities 160; 170 is further operative for plotting the calculated MAE and the calculated R2as a histogram. Further, the calculating of the trustworthiness score based on the calculated MAE and the calculated R2is performed by analysing the histogram.

[0054] According to yet another embodiment, the one or more network entities 160; 170 is further operative for the determining of an expected entity future state by determining a plurality of expected entity future states for each of a plurality of scenarios and operative for the receiving of an expected digital twin future state by receiving a plurality of expected digital twin future states for each of the plurality of scenarios, and operative for determining the trustworthiness score based on the plurality of expected entity future states and the plurality of expected digital twin future states.

[0055] According to still another embodiment, the one or more network entities 160; 170 further comprises, or is communicatively connected to, a database. The one or more network entities 160; 170 is further operative for obtaining the historical data associated with historical states of the at least part 130 of the communication network 100 at the one or more time points during a time period, before the obtaining of the current data associated with the current state, and storing the obtained historical data in the database.

[0056] According to still another embodiment, the one or more network entities 160; 170 is further operative for informing the concerned party of the determined expected entity future state at the future time point, and receiving, after or at the future time point, an indication associated with a correctness of the determinedexpected entity future state at the future time point in relation to an observed actual state at the future time point.

[0057] According to other embodiments, the one or more network entities 160; 170 may further comprise a communication unit 602, which may be considered to comprise conventional means for communication in the communication network. The instructions executable by said processing circuitry 603 may be arranged as a computer program 605 stored e.g. in said memory 604. The processing circuitry 603 and the memory 604 may be arranged in a sub-arrangement 601. The subarrangement 601 may be a micro-processor and adequate software and storage therefore, a Programmable Logic Device, PLD, or other electronic component(s) / processing circuit(s) configured to perform the methods mentioned above. The processing circuitry 603 may comprise one or more programmable processor, application-specific integrated circuits, field programmable gate arrays or combinations of these adapted to execute instructions.

[0058] The computer program 605 may be arranged such that when its instructions are run in the processing circuitry 603, the instructions cause the one or more network entities 160; 170 to perform the steps described in any of the described embodiments of the one or more network entities 160; 170 and its method. The computer program 605 may be carried by a computer program product connectable to the processing circuitry 603. The computer program product may be the memory 604, or at least arranged in the memory. The computer program product may be called a computer-readable storage medium 606. The memory 604 may be realized as for example a Random-access memory (RAM), Read-Only Memory (ROM) or an Electrical Erasable Programmable ROM (EEPROM). In some embodiments, a carrier may contain the computer program 605. The carrier may be one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or computer readable storage medium. The computer-readable storage medium 606 may be e.g., a CD, DVD or flash memory, from which the program could be downloaded into the memory 604. Alternatively, the computer program 605 may be stored on a server or any other entity to which the one ormore network entities 160; 170 has access via the communication unit 602. The computer program 605 may then be downloaded from the server into the memory 604.

[0059] Although the description above contains a plurality of specificities, these should not be construed as limiting the scope of the concept described herein but as merely providing illustrations of some exemplifying embodiments of the described concept. It will be appreciated that the scope of the presently described concept fully encompasses other embodiments which may become obvious to those skilled in the art, and that the scope of the presently described concept is accordingly not to be limited. Reference to an element in the singular is not intended to mean "one and only one" unless explicitly so stated, but rather "one or more." All structural and functional equivalents to the elements of the abovedescribed embodiments that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed hereby. Moreover, it is not necessary for an apparatus or method to address each and every problem sought to be solved by the presently described concept, for it to be encompassed hereby. In the exemplary figures, a broken line generally signifies that the feature within the broken line is optional.

Claims

CLAIMS1. A method performed by one or more network entities (160; 170) for evaluating trustworthiness of a network digital twin (150) set up as a digital twin of at least a part (130) of a communication network (100), the method comprising:obtaining (204), from the at least part (130) of the communication network, (100) current data associated with a current state of the at least part (130) of the communication network (100),determining (206) an expected entity future state of the at least part (130) of the communication network (100) at a future time point, based on the obtained current data associated with the current state and on historical data associated with one or more historical states of the at least part (130) of the communication network (100) obtained at one or more time points during a time period,receiving (208), from the network digital twin (150), an expected digital twin future state of the at least part (130) of the communication network (100) at the future time point, wherein the expected digital twin future state is simulated by the network digital twin (150) based on the current data associated with the current state that the network digital twin (150) has received from the at least part (130) of the communication network (100),determining (210) a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, andselectively informing (212) a concerned party (140; 180) of the determined trustworthiness score (210).

2. The method according to claim 1 , wherein the one or more network entities (160; 170) comprises a belief network structure, and wherein the expected entity future state is determined (206) using the belief network structure.

3. The method according to claim 1 or 2, wherein the selectively informing (212) a concerned party (140; 180) of the determined trustworthiness score comprises only informing the concerned party (140; 180) when the determined trustworthiness score indicates a trustworthiness of the expected digital twin future state lower than a defined threshold.

4. The method according to any of the preceding claims, wherein the one or more network entities (160; 170) is situated in one or more network functions of a core network of the communication network (100) or in a service management and orchestration, SMO, network function of a radio access network, RAN (130), of the communication network (100).

5. The method according to any of the preceding claims, wherein the expected entity future state and the expected digital twin future state each comprises a plurality of state characteristics, wherein the trustworthiness score is determined (210) by comparing one or more state characteristics of the plurality of state characteristics of the expected entity future state with same one or more state characteristics of the plurality of state characteristics of the expected digital twin future state.

6. The method according to claim 5, wherein the comparing of the one or more state characteristics of the plurality of state characteristics of the expected entity future state with same one or more state characteristics of the plurality of state characteristics of the expected digital twin future state comprises:calculating a mean absolute error, MAE, of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state,calculating a coefficient of determination, R2, of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state, andcalculating the trustworthiness score based on the calculated MAE and the calculated R2.

7. The method according to claim 6, further comprising:plotting the calculated MAE and the calculated R2as a histogram, wherein the calculating of the trustworthiness score based on the calculated MAE and the calculated R2is performed by analysing the histogram.

8. The method according to any of the preceding claims, wherein the determining (206) of an expected entity future state comprises determining aplurality of expected entity future states for each of a plurality of scenarios and wherein the receiving (208) of an expected digital twin future state comprises receiving a plurality of expected digital twin future states for each of the plurality of scenarios, and wherein the trustworthiness score is determined (210) based on the plurality of expected entity future states and the plurality of expected digital twin future states.

9. The method according to any of the preceding claims, wherein the one or more network entities (160; 170) further comprises, or is communicatively connected to, a database, the method further comprising:obtaining (201) the historical data associated with historical states of the at least part (130) of the communication network (100) at the one or more time points during a time period, before the obtaining (204) of the current data associated with the current state, andstoring (202) the obtained historical data in the database.

10. The method according to any of the preceding claims, further comprising:informing (214) the concerned party of the determined (206) expected entity future state at the future time point, andreceiving (216), after or at the future time point, an indication associated with a correctness of the determined expected entity future state at the future time point in relation to an observed actual state at the future time point.

11. One or more network entities (160; 170) configured for evaluating trustworthiness of a network digital twin (150) set up as a digital twin of at least a part (130) of a communication network (100), the one or more network entities (160; 170) comprising a processing circuitry (603) and a memory (604), said memory containing instructions executable by said processing circuitry, whereby the one or more network entities (160; 170) is operative for:obtaining, from the at least part (130) of the communication network, (100) current data associated with a current state of the at least part (130) of the communication network (100),determining an expected entity future state of the at least part (130) of the communication network (100) at a future time point, based on the obtained current data associated with the current state and on historical data associated with one or more historical states of the at least part (130) of the communication network (100) obtained at one or more time points during a time period,receiving, from the network digital twin (150), an expected digital twin future state of the at least part (130) of the communication network (100) at the future time point, wherein the expected digital twin future state is simulated by the network digital twin (150) based on the current data associated with the current state that the network digital twin (150) has received from the at least part (130) of the communication network (100),determining a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, andselectively informing a concerned party (140; 180) of the determined trustworthiness score.

12. The one or more network entities (160; 170) according to claim 11 , wherein the one or more network entities (160; 170) comprises a belief network structure, and wherein the one or more network entities (160; 170) is operative to determine the expected entity future state using the belief network structure.

13. The one or more network entities (160; 170) according to claim 11 or 12, operative to perform the selectively informing of a concerned party (140; 180) of the determined trustworthiness score by only informing the concerned party (140; 180) when the determined trustworthiness score indicates a trustworthiness of the expected digital twin future state lower than a defined threshold.

14. The one or more network entities (160; 170) according to any of claims 11-13, wherein the one or more network entities (160; 170) is situated in one or more network functions of a core network of the communication network (100) or in a service management and orchestration, SMO, network function of a radio access network, RAN (130), of the communication network (100).

15. The one or more network entities (160; 170) according to any of claims 11-14, wherein the expected entity future state and the expected digital twin future state each comprises a plurality of state characteristics, wherein the one or more network entities (160; 170) is operative to determine the trustworthiness score by comparing one or more state characteristics of the plurality of state characteristics of the expected entity future state with same one or more state characteristics of the plurality of state characteristics of the expected digital twin future state.

16. The one or more network entities (160; 170) according to claim 15, operative to compare the one or more state characteristics of the plurality of state characteristics of the expected entity future state with the same one or more state characteristics of the plurality of state characteristics of the expected digital twin future state by:calculating a mean absolute error, MAE, of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state,calculating a coefficient of determination, R2, of the one or more state characteristics of the expected entity future state with the same one or more state characteristics of the expected digital twin future state, andcalculating the trustworthiness score based on the calculated MAE and the calculated R2.

17. The one or more network entities (160; 170) according to claim 16, further being operative for:plotting the calculated MAE and the calculated R2as a histogram, wherein the calculating of the trustworthiness score based on the calculated MAE and the calculated R2is performed by analysing the histogram.

18. The one or more network entities (160; 170) according to any of claims 11-17, operative for the determining of an expected entity future state by determining a plurality of expected entity future states for each of a plurality of scenarios and operative for the receiving of an expected digital twin future state by receiving a plurality of expected digital twin future states for each of the plurality of scenarios, and operative for determining the trustworthiness score based on theplurality of expected entity future states and the plurality of expected digital twin future states.

19. The one or more network entities (160; 170) according to any of claims 11-18, wherein the one or more network entities (160; 170) further comprises, or is communicatively connected to, a database, the one or more network entities (160; 170) being operative for:obtaining the historical data associated with historical states of the at least part (130) of the communication network (100) at the one or more time points during a time period, before the obtaining of the current data associated with the current state, andstoring the obtained historical data in the database.

20. The one or more network entities (160; 170) according to any of claims 11-19, further being operative for:informing the concerned party of the determined expected entity future state at the future time point, andreceiving, after or at the future time point, an indication associated with a correctness of the determined expected entity future state at the future time point in relation to an observed actual state at the future time point.

21. A computer program (605) comprising instructions, which, when executed by at least one processing circuitry of one or more network entities (160; 170), configured for evaluating trustworthiness of a network digital twin (150) set up as a digital twin of at least a part (130) of a communication network (100), causes the one or more network entities (160; 170) to perform the following steps:obtaining, from the at least part (130) of the communication network, (100) current data associated with a current state of the at least part (130) of the communication network (100),determining an expected entity future state of the at least part (130) of the communication network (100) at a future time point, based on the obtained current data associated with the current state and on historical data associated with one or more historical states of the at least part (130) of the communication network (100) obtained at one or more time points during a time period,receiving, from the network digital twin (150), an expected digital twin future state of the at least part (130) of the communication network (100) at the future time point, wherein the expected digital twin future state is simulated by the network digital twin (150) based on the current data associated with the current state that the network digital twin (150) has received from the at least part (130) of the communication network (100),determining a trustworthiness score of the expected digital twin future state based on the expected entity future state and the expected digital twin future state, andselectively informing a concerned party (140; 180) of the determined trustworthiness score.

22. A carrier containing the computer program (605) according to claim 21 , wherein the carrier is one of an electronic signal, an optical signal, a radio signal, an electric signal or a computer readable storage medium (606).

Images