First node, second node and methods performed thereby for handling input data to a first machine learning reinforcement model

By optimizing the frequency of input data selection and prioritizing authentic data using digital twins, the method enhances the efficiency and accuracy of machine learning reinforcement learning models in telecommunications networks, addressing latency and cost issues in existing technologies.

WO2026233601A1PCT designated stage Publication Date: 2026-11-12TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/058653
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-03-25
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing methods for handling input data to machine learning reinforcement learning models in telecommunications networks face challenges such as long latencies and high costs due to the computational complexity and latency of behavioral models, leading to inefficient network optimization.

Method used

A method is introduced to optimize the frequency of input data selection for machine learning reinforcement learning models by determining how often to obtain input from behavioral models versus using previous states, prioritizing authentic data based on authenticity differences, and employing gradient descent with digital twins to enhance optimization efficiency and accuracy.

Benefits of technology

This approach improves the performance and convergence of network optimization by reducing latency and computational costs, enabling more efficient and accurate decision-making in telecommunications networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026058653_12112026_PF_FP_ABST
    Figure EP2026058653_12112026_PF_FP_ABST
Patent Text Reader

Abstract

A method performed by a first node (111). The first node (111) determines (304) a first frequency with which to select first data output by training a first ML RL model with one or more third nodes (113), to be used as input data to a second node (112) for training the first ML RL model, or use instead second data previously output by the one or more third nodes (113) in one or more previous iterations of the training of the first ML RL model. Each of the one or more third nodes (113) runs a behavioral model as a digital twin of one or more variables of a communications system (101), used to train the first ML RL model. The first ML RL model is to optimize a performance of the communications system (101). The first node (111) then outputs (305) the first frequency to the second node (112).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] FIRST NODE, SECOND NODE AND METHODS PERFORMED THEREBY FOR HANDLING INPUT DATA TO A FIRST MACHINE LEARNING REINFORCEMENT MODEL

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to a first node and methods performed thereby for handling input data to a first machine learning (ML) reinforcement learning (RL) model. The present disclosure further relates generally to a second node and methods performed thereby, for handling input data to a first ML RL model. The present disclosure also relates generally to computer programs and computer-readable storage mediums, having stored thereon the computer programs to carry out these methods.

[0004] BACKGROUND

[0005] Computer systems in a communications network or communications system may comprise one or more nodes. A node may comprise one or more processors which, together with computer program code may perform different functions and actions, a memory, a receiving port, and a sending port. A node may be, for example, a server. Nodes may perform their functions entirely on the cloud.

[0006] Computer systems may be comprised in a telecommunications network. The telecommunications network may cover a geographical area which may be divided into cell areas, each cell area being served by a type of node, a network node in the Radio Access Network (RAN), radio network node or Transmission Point (TP), for example, an access node such as a Base Station (BS), e.g., a Radio Base Station (RBS), which sometimes may be referred to as e.g., gNB, evolved Node B (“eNB”), “eNodeB”, “NodeB”, “B node”, or Base Transceiver Station (BTS), depending on the technology and terminology used. The base stations may be of different classes such as e.g., Wide Area Base Stations, Medium Range Base Stations, Local Area Base Stations and Home Base Stations, based on transmission power and thereby also cell size. A cell may be understood to be the geographical area where radio coverage may be provided by the base station at a base station site. One base station, situated on the base station site, may serve one or several cells. Further, each base station may support one or several communication technologies. The telecommunications network may also comprise network nodes which may serve receiving nodes, such as user equipments, with serving beams.

[0007] The standardization organization Third Generation Partnership Project (3GPP) is currently in the process of specifying a New Radio Interface called Next Generation Radio or New Radio (NR), as well as a Fifth Generation (5G) Packet Core Network, which may be referred to as 5G Core Network (5GC). The advantages of 5G NR may include higherbandwidth, more resources, low latency and network slicing. 5G may provide services to various applications, such as enhanced Mobile Broad Band (eMBB), machine to Machine type communication (mMTC), Ultra Reliable Low Latency Communication (URLLC), etc.

[0008] 5G may be understood to bring in sizeable flexibility with technological advancements along with innovations of cloud and Al. This may be understood to bring a whole new set of opportunities in the enterprise segment.

[0009] For many enterprises, mobile cellular technology has already proven to bring great value to their digitalization process, which may include numerous use cases, such as autonomous robotics, enhanced video services, connected vehicles, remote operations, hazard, and maintenance sensors etc. This may be understood to not only enhance productivity in connected factories, but also make workplaces safer.

[0010] In the course of operations of the telecommunications network, data may be collected via the telecommunications network, which may enable to monitor and manage different functions.

[0011] The advent of for example, the Internet of Things (loT) has exponentially increased the amount of data to be monitored. The availability of large amounts of data, such as those collected for example, from loT devices, may be understood to enable the possibility of analyzing such data to make predictions on events, with a high predictive power. To make predictions on events may be understood to refer to building mathematical models that may fit those data, which mathematical models may then be used to make predictions for such events. Within this context, machine learning models may be used to analyze the data collected, and enable an improved management of different types of operations via the telecommunications network.

[0012] Machine Learning

[0013] Machine learning (ML) may be understood as the study of computer algorithms that may improve automatically through experience. It is seen as a part of Artificial Intelligence (Al). ML algorithms may build a model based on sample data, known as "training data", in order to make predictions or decisions without being explicitly programmed to do so. ML algorithms may be used in a wide variety of applications, such as email filtering and computer vision, where it may be difficult or unfeasible to develop conventional algorithms to perform the needed tasks.

[0014] There may be basically 3 types of ML Algorithms: Supervised Learning, Unsupervised Learning, and Reinforcement Learning (RL).

[0015] Supervised Learning algorithms may comprise a target / outcome variable, or dependent variable, which may have to be predicted from a given set of predictors, that is, independent variables. Using this set of variables, a function may be generated that may map inputs to desired outputs. The training process may continue until the model may achieve a desiredlevel of accuracy on the training data. Once an ML model may have been trained, an inference process may begin, whereby new data may be run through the ML model to calculate an output. Examples of Supervised Learning may be Regression, Decision Tree, Random Forest, KNN, Logistic Regression etc.

[0016] In Unsupervised Learning algorithms, there may be no target or outcome variable to predict / estimate. It may be used for clustering a population into different groups, which may be widely used for segmenting customers in different groups for specific intervention. Examples of Unsupervised Learning may be K-means, mean-shift clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Expectation-Maximization (EM) Clustering using Gaussian Mixture Models (GMM), Agglomerative Hierarchical Clustering, etc....

[0017] Cluster analysis or clustering may be understood as an ML technique which may comprise grouping a set of objects in such a way that objects in the same group, which may be called a cluster, may be understood to be more similar, in some sense, to each other than to those in other groups, that is, other clusters. It may be understood as a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including pattern recognition, image analysis, information retrieval, bioinformatics, data compression, computer graphics and ML.

[0018] Using an RL algorithm, a machine may be trained to make specific decisions. It may be understood to work as follows: the machine may be exposed to an environment where it may train itself continually using trial and error. This machine may learn from past experience and may try to capture the best possible knowledge to make accurate decisions. An example of RL may be a Markov Decision Process (MDP). The training using RL may comprise generating an ML model. To train such an ML model, an agent, given a state of the environment, may take an action in this environment and receive a reward. The action may result in a new state of the environment. This process may be repeated in a loop. Over time, the agent may learn to take actions that may result in larger immediate and future rewards, meaning that it may be understood to be in the best interest of the agent not to take the action that may only lead to the highest reward in the next state, but the action that may cumulatively lead to the highest reward in the next state and in a future number of states.

[0019] The agent may comprise a neural network which may input the state and may produce an action. There may be several ML algorithms that may be used for training the network of the agent, e.g., policy-learning based, such as actor-critic approaches, or value-based learning, such as deep-q networks.

[0020] To optimize the configuration of a telecommunications network and ensure high-quality recommendations, an Al-based optimizer may be employed, leveraging either RL or mathematical optimization algorithms such as Alternating Direction Method of Multipliers (ADMM). An Al-based optimizer may be understood as an algorithm capable of improving thenetwork performance with a suitable selection of one or more network configuration parameters, which may be determined by relying totally or partially on Al technique. However, due to challenges of interacting directly with real-world systems, the Al optimizer may face potential issues with unwanted intermediate states and long execution times, e.g., when interacting with external data sources which may be implemented in the form of digital twins or behavioural model that may require significant time either to observe or estimate the state of the network. An intermediate state may be understood as a stage of the optimization process in which the network may have already been reconfigured, but may not have yet reached its maximum performance potential. To address this, industry experts have taken the initiative to develop a digital twin (DT), serving as a partial virtual replica of the telecommunications system. A digital twin may be understood as a virtual representation of a system that may be used to simulate and analyze real-world scenarios. This DT may be understood to provide a safe and authentic environment for the Al-based optimizer to interact with and perform its optimization tasks. The environment may be understood to be safe because it may allow for experimentation, testing, and analysis without posing risks to the actual physical entity or system. The environment may be understood to be authentic because it may provide a highly accurate and detailed representation of its physical counterpart, enabling reliable simulations and analyses.

[0021] Taking the example of the RAN DT [1], the process of creating and / or modelling a RAN DT may involve constructing multiple behavior models (BMs) to simulate the complex interactions between user equipments (UEs) and base stations (BSs). These BMs may be understood to play a relevant role in providing input to the optimizer. As a summarized view, the optimizer may utilize data and insights from these BMs to recommend tuning certain parameters within the network. For instance, the optimizer may suggest optimal configurations for parameters such as antenna tilt, transmission power, and other relevant options.

[0022] It may be important to note that executing these BMs may be computationally expensive, depending on the analysis requirements and the chosen implementation. However, the collaboration between the BMs and the optimizer / agent may be understood to be relevant in achieving efficient and effective network optimization. Figure 1 is a schematic diagram illustrating a high level simplified overview of this collaborative process between an Al optimizer 10, with data represented in Figure 1 by boxes 11, and multiple computation BMs 12, BM_1, BM_2, BM_3 ..., denoted by arrows. The data represented by the boxes 11 may be understood to be data collected by the network in aggregation which is referred to as Records of Period (ROPs). Typically this data may be raw. The data, as depicted in Figure 1, may comprise data on cells 13 and data on UEs 14. For each of these groups, considering a static DT timespan 15, which is represented in Figure 1, in the vertical axis, the data 11 may be divided in different ROPs 16: ROP(1),..., ROP(N-1), and ROP(N). The horizontal axisrepresents the execution of the BMs. The observations of the network may be understood to be too complex to be estimated using a formula with new Configuration Management (CM) parameter values. For example, pathloss for a number of UEs connected to a base station may be costly to estimate since it may be understood to require a lot of signalling to each UE, which may be understood to take time. It may also take a lot of time to estimate using machine learning models, since the estimation may need to account for reflections of beams and also knowledge of the environment. The only way to get the outcome may be to execute the BMs sequentially or in parallel depending on corresponding BM dependencies. That is, instead of asking UEs to calculate pathloss, it may be possible to use a model, or a BM, but this may be also understood to be costly.

[0023] Figure 1 may be understood to illustrate a dynamic RAN DT 17 in action as it may receive CM parameter changes 18 proposed by the Al optimizer 10, and may output new values 19 for cell-level synthetic counters, such as coverage counters 20, inference counters 21, energy consumption counters 22, etc.. In the form of reinforcement learning based optimizer, these counters may then be used to construct the state and reward of the RL agent. In the form of mathematical based optimizer, these counters may be used to calculate the cost function value. Both cases may be understood to require many Optimizer 10-DT 17 iterations to search the global optimal CM parameter settings.

[0024] Examples of behavior model may be Reference Signal Received Power (RSRP) update for all measurement reports, synthetic handover for UEs, Signal to Inference plus noise ratio (SI NR) update, Top neighboring cell identification, user experience update, cell utilization update, cell energy consumption update, etc..

[0025] In existing methods, RL methodology may be used to compute efficient actuations. However, existing methods to compute actuations may involve long latencies and high costs.

[0026] SUMMARY

[0027] According to the foregoing, it is an object of embodiments herein to improve the handling of input data to an ML RL model.

[0028] According to a first aspect of embodiments herein, the object is achieved by a computer-implemented method, performed by a first node. The method is for handling input data to a first ML RL model. The first node operates in a computer system. The first node determines a first frequency with which to select first data. The first data is output by training the first ML RL model with one or more third nodes operating in the computer system. The first node determines the frequency with which to select the first data to be used as input data to a second node operating in the computer system, for training the first ML RL model, or use instead second data previously output by the one or more third nodes. The second data has been previously output by the one or more third nodes in one or more previous iterations of thetraining of the first ML RL model. Each of the one or more third nodes run a respective behavioral model as a digital twin of one or more variables of the communications system. The one or more variables are used to train the first ML RL model by the second node. The first ML RL model is to optimize a performance of the communications system. The first node then outputs the determined first frequency to the second node.

[0029] According to a second aspect of embodiments herein, the object is achieved by a computer-implemented method, performed by the second node. The method is for handling the input data to the first ML RL model. The second node operates in the computer system. The second node trains the first ML RL model with the one or more third nodes operating in the computer system for a second plurality of iterations. The first ML RL model is to optimize the performance of the communications system. The training comprises, for every iteration of the second plurality of iterations obtaining, from the first node operating in the computer system, the first frequency. The first frequency is with which to select the first data output by training the first ML RL model with the one or more third nodes operating in the computer system. The first frequency is with which to select the first data to be used as input data to the second node fortraining the first ML RL model, or use instead the second data previously output by the one or more third nodes in the one or more previous iterations of the training of the first ML RL model. Each of the one or more third nodes run the respective behavioral model as the digital twin of the one or more variables of the communications system. The one or more variables are used to train the first ML RL model by the second node. The training further comprises selecting one of the first data and the second data as input data for training the first ML RL model based on the obtained first frequency.

[0030] According to a third aspect of embodiments herein, the object is achieved by the first node. The first node may be understood to be for handling the input data to the first ML RL model. The first node is configured to operate in the computer system. The first node is configured to determine the first frequency with which to select the first data output by training the first ML RL model with the one or more third nodes configured to operate in the computer system, to be used as input data to the second node configured to operate in the computer system, for training the first ML RL model, or use instead the second data previously output by the one or more third nodes in the one or more previous iterations of the training of the first ML RL model. Each of the one or more third nodes is configured to run the respective behavioral model as the digital twin of the one or more variables of the communications system. The one or more variables are configured to be used to train the first ML RL model by the second node. The first ML RL model is configured to optimize the performance of the communications system. The first node is also configured to output the first frequency configured to be determined to the second node.According to a fourth aspect of embodiments herein, the object is achieved by the second node. The second node may be understood to be for handling the input data to the first ML RL model. The second node is configured to operate in the computer system. The second node is further configured to train the first ML RL model with the one or more third nodes configured to operate in the computer system for the second plurality of iterations. The first ML RL model is configured to optimize the performance of the communications system. The training is configured to comprise, for every iteration of the second plurality of iterations: a) obtaining, from the first node configured to operate in the computer system, the first frequency with which to select the first data output by training the first ML RL model with the one or more third nodes configured to operate in the computer system, to be used as input data to the second node, fortraining the first ML RL model, or use instead the second data previously output by the one or more third nodes in the one or more previous iterations of the training of the first ML RL model; Each of the one or more third nodes is configured to run the respective behavioral model as the digital twin of the one or more variables of the communications system 101; the one or more variables are configured to be used to train the first ML RL model by the second node, and b) selecting one of the first data and the second data as input data for training the first ML RL model based on the first frequency configured to be obtained.

[0031] According to a fifth aspect of embodiments herein, the object is achieved by a computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the first node.

[0032] According to a sixth aspect of embodiments herein, the object is achieved by a computer-readable storage medium, having stored thereon the computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the first node.

[0033] According to a seventh aspect of embodiments herein, the object is achieved by a computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the second node.

[0034] According to an eighth aspect of embodiments herein, the object is achieved by a computer-readable storage medium, having stored thereon the computer program, comprising instructions which, when executed on at least one processing circuitry, cause the at least one processing circuitry to carry out the method performed by the second node.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Examples of embodiments herein are described in more detail with reference to the accompanying drawings, according to the following description.

[0036] Figure 1 is a schematic diagram illustrating a high level overview of a digital twin of a network, including an Al optimizer and multiple BMs.

[0037] Figure 2 is a schematic diagram illustrating two non-limiting examples, in panels a) and b), of a computer system, according to embodiments herein.

[0038] Figure 3 is a flowchart depicting a method in a first node, according to embodiments herein. Figure 4 is a flowchart depicting a method in a computer system, according to embodiments herein.

[0039] Figure 5 is a schematic diagram depicting a non-limiting example of a method in a computer system performed according to embodiments herein, in a) a training phase, and b) an execution phase.

[0040] Figure 6 is a schematic diagram depicting particular aspects of a non-limiting example of the method performed by a first node, according to embodiments herein.

[0041] Figure 7 is a schematic diagram depicting particular aspects of a non-limiting example of the method performed according to embodiments herein.

[0042] Figure 8 is a schematic diagram depicting other aspects of the non-limiting example of the method performed according to embodiments herein.

[0043] Figure 9 is a schematic diagram depicting a non-limiting example of a computer system according to embodiments herein.

[0044] Figure 10 is a schematic block diagram illustrating an embodiment of a first node, according to embodiments herein.

[0045] Figure 11 is a schematic block diagram illustrating an embodiment of a computer system, according to embodiments herein.

[0046] DETAILED DESCRIPTION

[0047] As part of the development of embodiments herein, one or more problems with the existing technology will first be identified and discussed.

[0048] The challenge with a setup and / or approach such as that depicted and described in relation to Figure 1 may be understood to be that, as the number of BMs increases, so does the input to the Al-based optimizer. That input to the optimizer may be viewed as a dependency, since the optimizer may not be able to produce a decision in the absence of that input originating from corresponding BMs. One or more BMs may work together towards estimating some aspect of the state of the network. However, in a real-life setup, such input may take a long time to be obtained due to network latency, computation and / or processingcomplexity of the BM, software errors in the implementation of the BMs, but also hardware errors in any sensors that may be employed by the BMs.

[0049] In the state of the art, data augmentation techniques or models may be encountered which may be developed using centralized or distributed approaches, such as federated or split learning. Such techniques may be used to train models collaboratively thus overcoming the problem of accessing missing data in one instance of a BM by learning that data indirectly from another node, if such data is available. Still the underlying issue related to data collection remains, as it is likely that needed data that may be required from a specific source may not be available in any other source, hence delaying input to the optimizer. Moreover, data augmentation techniques may be problematic as it may be hard to assess if generated data may be genuine or not.

[0050] In offline reinforcement learning [2], importance sampling may be understood to be a known method which may be used to address these challenges as well, where samples may be weighted based on the effect they may have on the policy to be learned. However, that approach may be understood to be limited to offline learning. This may mean that collection of data may extend over long periods of time, before training of the data may begin.

[0051] Methods on prioritized experience replay in Deep Q Network (DQN) training also exist [3], but they may be understood to typically prioritize experience tuples based on the magnitude of their temporal-difference (TD) error.

[0052] Certain aspects of the present disclosure and their embodiments address the challenges identified with the existing methods and provide solutions to the challenges discussed.

[0053] To overcome the challenges just described, embodiments herein may be understood to provide a method to optimize decisions in mobile network digital twins with delay tolerance.

[0054] More particularly, embodiments herein may provide a method to identify how frequently input to the optimizer may need to be obtained by a BM versus using associated input from a previous state, and the impact that may have to the performance of the optimizer.

[0055] Embodiments herein may be understood to deal with an online learning setting environment which may be understood to mean that the distribution of samples in advance may not be known, since in online learning the sample distribution may only be known once data may have been collected and may be different from one batch to another.

[0056] In contrast to the methods on prioritized experience replay in DQN training, which may typically prioritize experience tuples based on the magnitude of their Temporal Difference (TD) error, examples of the approach described herein may be understood to focus on prioritizing experience tuples based on the authenticity of the data, which may be determined by the difference between authentic, e.g., step = nN and nN+1, and less-authentic tuples, e.g., other steps.This distinction may be understood to set the method of embodiments herein apart and may introduce a novel way of enhancing DQN optimizer training, which may be based on a gradient descent, with DT. The same principle may be extended to mathematical-based optimization processes, such as e.g., Alternating Direction Method of Multipliers (ADMM). In this setup, the optimizer may be enabled to directly minimize the target cost function without the need for state information. Block coordinate descent may be understood to be a multivariate function optimizer that may choose a single direction and may optimize towards that direction, which may be more efficient than other approaches, since it may not explore all possible directions; however it may not find the true minimum. Notably, contemporary practices, such as the block coordinate descent algorithm, may attempt to estimate the gradient of the unknown BM function by invoking the function. At step = nN, the gradient estimation value may be trusted, as it may accurately compute all the BMs. By employing the authenticity-based prioritization of data in this optimization approach, embodiments herein may enable to enhance the overall efficiency and accuracy of the optimization process, resulting in improved performance and convergence.

[0057] Some of the embodiments contemplated will now be described more fully hereinafter with reference to the accompanying drawings, in which examples are shown. In this section, the embodiments herein will be illustrated in more detail by a number of exemplary embodiments. Other embodiments, however, are contained within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art. It should be noted that the exemplary embodiments herein are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments.

[0058] Several embodiments and examples are comprised herein. It should be noted that the embodiments and / or examples herein are not mutually exclusive. Components from one embodiment or example may be tacitly assumed to be present in another embodiment or example and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments and / or examples.

[0059] Figure 2 depicts two non-limiting examples, in panels “a” and “b”, respectively, of a computer system 100, in which embodiments herein may be implemented. In some example implementations, such as that depicted in the non-limiting examples of Figure 2, the computer system 100 may be a computer network. In other example implementations, such as that depicted in panel b) of Figure 2, the computer system 100 may be implemented in, or may be able to communicate with or integrate with, a communications system 101, sometimes also referred to as a telecommunications system, cellular radio system, cellular network or wirelesscommunications system. In some examples, the communications system 101 may comprise network nodes which may serve receiving nodes, such as wireless devices, with serving beams.

[0060] In some examples, the communications system 101 may be, for example, a communications network, such as 5G system, or Next Gen network. In other examples, the communications system 101 may be a newer system, e.g., a Sixth Generation (6G) system, with similar functionality. Yet in other examples, the communications system 101 may additionally, or alternatively, support other technologies, such as LTE, e.g., LTE Frequency Division Duplex (FDD), LTE Time Division Duplex (TDD), LTE Half-Duplex Frequency Division Duplex (HD-FDD), LTE operating in an unlicensed band, Wideband Code Division Multiple Access (WCDMA), Universal Terrestrial Radio Access (UTRA) TDD, Global System for Mobile communication (GSM) / Enhanced Data Rate for GSM Evolution (EDGE) Radio Access Network (GERAN) network, Ultra-Mobile Broadband (UMB), EDGE network, network comprising of any combination of Radio Access Technologies (RATs) such as e.g., MultiStandard Radio (MSR) base stations, multi-RAT base stations etc., any 1 rd Generation Partnership Project (3GPP) cellular network, Wireless Local Area Network / s (WLAN) or WiFi network / s, Worldwide Interoperability for Microwave Access (WiMax), Internet of Things (loT), Machine Type Communication (MTC), IEEE 802.15.4-based low-power short-range networks such as IPv6 over Low-Power Wireless Personal Area Networks (6LowPAN), Zigbee, Z-Wave, Bluetooth Low Energy (BLE), or any cellular network or system.

[0061] The computer system 100 may comprise a plurality of nodes, whereof a first node 111 and a second node 112 are depicted in Figure 2. In some embodiments, the computer system 100 may comprise one or more third nodes 113. In some embodiments, the computer system 100 may comprise a fourth node 114. The computer system 100 may comprise additional nodes.

[0062] Any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may be understood, respectively, as a first computer system or server, a second computer system or server, one or more third computer systems or servers and a fourth computer system or server. Any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may be implemented as a standalone server in e.g., a host computer in the cloud 115, as depicted in the non-limiting example of Figure 2b) for the second node 112. Any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may be a distributed node or distributed server, such as a virtual node in the cloud 115, and may perform some of its respective functions locally, e.g., by a client manager, and some of its functions in the cloud 115, by e.g., a server manager. In other examples, any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may perform its functions entirely on the cloud 115, orpartially, in collaboration or collocated with a radio network node. Yet in other examples, any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may also be implemented as processing resources in a server farm. In yet other examples, any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114 may be comprised in a device, such as the device 140 described below, or on the edge. Any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may be under the ownership or control of a service provider or may be operated by the service provider, or on behalf of the service provider.

[0063] In some examples, any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may be co-located or be the same node.

[0064] Any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114 may have a capability to perform machine-implemented learning procedures, which may be also referred to as “machine learning” (ML).

[0065] Any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114 may have a capability to manage an artificial neural network. The artificial neural network may be understood as a machine learning framework, which may comprise a collection of connected nodes, where in each node or perceptron, there may be an elementary decision unit. Each such node may have one or more inputs and an output. The input to a node may be from the output of another node or from a data source. Each of the nodes and connections may have certain weights or parameters associated with it. In order to solve a decision task, the weights may be learnt or optimized over a data set which may be representative of the decision task. The most commonly used node may have each input separately weighted, and the sum may be passed through a non-linear function which may be known as an activation function. The nature of the connections and the node may determine the type of the neural network, for example a feedforward network, recurrent neural network etc. That any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may have the capability to manage the artificial neural network may be understood herein as having the capability to store the training data set and the models that may result from the machine learning, to train a new model, and once the model may have been trained, to use this model for prediction. In some embodiments, the system that may be used for training the model and the one used for prediction may be different. In some embodiments, the first node 111 may be understood as a node having a capability to use a model, once the model may have already been trained. However, in other embodiments, the first node 111 may also have a capability to train the artificial neural network .

[0066] Any of the second node 112, the one or more third nodes 113 and the fourth node 114, and in some embodiments, the first node 111, used fortraining the artificial neural network may require more computational resources than the first node 111 that may use the trainedmodel to make predictions. Therefore, any of the second node 112, the one or more third nodes 113 and the fourth node 114, and in some embodiments, the first node 111, used for training the artificial neural network may, for example, support running python / Java with T ensorflow or Pytorch, Theano etc... Any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may also have GPU capabilities.

[0067] In some embodiments, any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114 may be a core network node, such as, e.g., a network data analytics function (NWDAF), a Service management and orchestration (SMO) node, a positioning node, a coordinating node, a Self-Optimizing / Organizing Network (SON) node, a Minimization of Drive Test (MDT) node, etc.... In 5G, for example, any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114 may be located in the Operations Support Systems (OSS).

[0068] In other examples, any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may be a radio network node, such as radio network node 130, depicted in Figure 2 b). The radio network node 130 may be, e.g., comprised in a Radio Access Network of the telecommunications system. That is, the radio network node 130 may be a transmission point such as a radio base station, for example a gNB, an eNB, or any other network node with similar features capable of serving a wireless device, such as a user equipment (UE) or a machine type communication device, in the computer system 100. In typical examples, the radio network node 130 may be a base station, such as a gNB or an eNB. In other examples, the radio network node 130 may be a distributed node, such as a virtual node in the cloud 115, and may perform its functions entirely on the cloud 115, or partially, in collaboration with a radio network node.

[0069] Any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114, in some examples, may be, or comprise, a central unit (CU), e.g., a CU control plane (CU-CP).

[0070] In some examples, the computer system 100 may include an access network, such as a radio access network (RAN), and a core network, which may include one or more core network nodes. The access network may include one or more access network nodes, such as any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114, e.g., which may be generally referred to as network nodes, or any other similar 3rd Generation Partnership Project (3GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node may not necessarily be limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it may be understood that network nodes may include disaggregated implementations or portions thereof. For example, in some embodiments, the computer system 100 may include one or more Open-RAN (ORAN) network nodes. An ORANnetwork node may be understood to be a node in the computer system 100 that may support an ORAN specification, e.g., a specification published by the O-RAN Alliance, or any similar organization, and may operate alone or together with other nodes to implement one or more functionalities of any node in the computer system 100, including one or more network nodes and / or core network nodes.

[0071] Examples of an ORAN network node may include an open radio unit (0-Rll), an open distributed unit (0-Dll), an open central unit (O-CU), including an O-CU control plane (O-CU-CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller, near-real time or non-real time, hosting software or software plug-ins, such as a near-real time control application, e.g., xApp, or a non-real time control application, e.g., rApp, or any combination thereof, the adjective “open” designating support of an ORAN specification. Any of the first node 111 , the second node 112, the one or more third nodes 113 and the fourth node 114 may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1, F1, W1, E1, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment, in which one or more network functions may be virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an O-2 interface defined by the O-RAN Alliance or comparable technologies.

[0072] The communications system 101 may cover a geographical area, which in some embodiments may be divided into cell areas, wherein each cell area may be served by a radio network node 130, although, one radio network node 130 may serve one or several cells. In the example of panel b) of Figure 2, the radio network node 130 serves a cell 120, although this may be understood to be for illustrative purposes only. The radio network node 130 may serve additional cells. The radio network node 130 may be of different classes, such as, e.g., macro eNodeB, home eNodeB or pico base station, based on transmission power and thereby also cell size. In some examples, the radio network node 130 may serve receiving nodes with serving beams. The radio network node 130 may be directly connected to one or more core networks.

[0073] Any of the first node 111, the second node 112, the one or more third nodes 113 and the fourth node 114, and / or any of the nodes comprised in the computer system 100 may support one or several communication technologies, and its name may depend on the technology and terminology used.

[0074] A device 140 may be comprised in the communications system 101. The device 140 comprised in the computer system 100 may be a wireless communication device such as a 5G UE, or a UE, which may also be known as e.g., mobile terminal, wireless terminal and / ormobile station, a Customer Premises Equipment (CPE) a mobile telephone, cellular telephone, or laptop with wireless capability, just to mention some further examples. The device 140 comprised in the telecommunications system may be, for example, portable, pocket-storable, hand-held, computer-comprised, or a vehicle-mounted mobile device, enabled to communicate voice and / or data, via the RAN, with another entity, such as a server, a laptop, a Personal Digital Assistant (PDA), or a tablet, Machine-to-Machine (M2M) device, device equipped with a wireless interface, such as a printer or a file storage device, modem, sensor, loT device, or any other radio network unit capable of communicating over a radio link in a communications system. The device 140 comprised in the telecommunications system may be enabled to communicate wirelessly in the telecommunications system. The communication may be performed e.g., via a RAN, and possibly the one or more core networks, which may be comprised within the telecommunications system.

[0075] It may be understood that the communications system 101 may comprise additional radio network nodes 130 and / or additional devices 140.

[0076] The first node 111 may be configured to communicate within the computer system 100 with the fourth node 114 over a first link 141 , e.g., a radio link, or a wired link. The first node 111 may be configured to communicate within the computer system 100 with the one or more third nodes 113 over one or more second links 142, e.g., radio links, or wired links. The one or more third nodes 113 may be configured to communicate within the computer system 100 with the second node 112 over one or more third links 143, e.g., radio links, or wired links. The second node 112 may be configured to communicate within the computer system 100 with the radio network node 130 over a fourth link 144, e.g., a radio link, or a wired link. The device 140 may be configured to communicate within the computer system 100 with the radio network node 130 over a fifth link 145, e.g., a radio link, or a wired link.

[0077] Any of the links just described may be a direct link or may be comprised of a plurality of individual links, wherein it may go via one or more computer systems or one or more core networks in the computer system 100, which are not depicted in Figure 2, or it may go via an optional intermediate network. The intermediate network may be one of, or a combination of more than one of, a public, private or hosted network; the intermediate network, if any, may be a backbone network or the Internet; in particular, the intermediate network may comprise two or more sub-networks, which is not shown in Figure 2.

[0078] In general, the usage of “first”, “second”, “third”, “fourth”, and / or “fifth” herein may be understood to be an arbitrary way to denote different elements or entities, and may be understood to not confer a cumulative or chronological character to the nouns they modify.

[0079] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein, the disclosed subject matter should not beconstrued as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0080] Embodiments of a computer-implemented method, performed by the first node 111, will now be described with reference to the flowchart depicted in Figure 3. The method is for handling input data to a first machine learning (ML) reinforcement learning (RL) model. The first node 111 operates in the computer system 100.

[0081] Several embodiments are comprised herein. In some embodiments all the actions may be performed. In some embodiments, some actions may be optional. In Figure 3, optional actions are indicated with dashed lines. It should be noted that the examples herein are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description.

[0082] Action 301

[0083] In this Action 301 , the first node 111 may obtain, for every iteration of a first plurality of iterations, a respective first indication of a state of the communications system 101.

[0084] The state may comprise a configuration of one or more parameters, normalized average and standard deviation of synthetic counters. The state of the communications system 101 may include parameters, such as number of Radio Resource Control (RRC) connections, throughput, latency for every cell of the network, transmission power, energy consumption, etc.

[0085] The respective first indication may comprise one or more respective values of the one or more parameters of the communications system 101.

[0086] The one or more parameters may be, for example, maximum (Max) transmission (tx) power, RET, frequency, antenna height, inter site distance, offered traffic per cell, etc..

[0087] The one or more respective values of the one or more parameters may be, for example, maximum (Max) transmission (tx) power: [37, 50] dBm, RET: [0,14] degrees, frequency: {0.7, 0.8, 0.9, 1.8. 1.9, 2.1, 2.6} GHz, antenna height: [20, 30] meters (m), inter site distance: [400, 1500] m, offered traffic per cell: [1, 10] MHz.

[0088] The one or more respective values may be values obtained from the one or more third nodes 113 operating in the computer system 100. Each of the one or more third nodes 113 may run a respective behavioral model (BM) as a digital twin of one or more variables of the communications system 101.The one or more variables may be used to train a first ML RL model by the second node 112 operating in the computer system 100. The first ML RL model may be to optimize a performance of the communications system 101. That is, one or more aspects of the performance of the communications system 101. The second node 112 may be an Al optimizer. The second node 112 may run a Double Deep Q Network (DDQN) algorithm.

[0089] The one or more variables may be any of pathloss, RSRP measurements, energy consumption, transmission power, network KPIs such as latency, throughput, etc. The one or more variables may comprise counters, e.g., coverage counters, interference counters, energy consumption counters, etc..

[0090] In some examples, the respective first indication may indicate a list of parameter changes tunable inside the one or more third nodes 113, e.g., the digital twin.

[0091] In some examples, the respective first indication may indicate a delta parameter change requested by the second node 112, that is, the Al optimizer.

[0092] In some embodiments, the one or more respective values may be outdated values obtained from one or more third nodes 113 operating in the computer system 100.

[0093] Outdated values may be understood to mean that they may have been obtained in one or more previous iterations to a particular iteration of the first plurality of iterations. That is, values that may not be directly observed, e.g., due to lack of observability. However, these values may be computed. As the computation may be understood to be costly, it may unlikely that it may be performed often, which may result in the values being outdated. The outdated values may be, for example, previous normalized synthetic counters in the state.

[0094] In some examples, the respective first indication may indicate a list of M previous synthetic counters values produced by a particular third node of the one or more third nodes 113, that is, a particular BM.

[0095] In some examples, the respective first indication may indicate a list of M previous synthetic counters values produced by previous one or more third nodes 113, that is, previous BMs, that may be related to the one that may be under study, for example, if congestion rate is being estimated, it may be impacted by the average RSRP and SINR.

[0096] Obtaining in this Action 301 may comprise receiving from another node in the computer system 100, for example, from the device 140, via the radio network node 130, or fetching or retrieving from a memory.

[0097] Action 302

[0098] In this Action 302, the first node 111 may train, for the first plurality of iterations, a second machine learning model (MLM). The second MLM may be to determine a frequency with which to select first data from the one or more third nodes 113 to be used as input data to the second node 112 for training the first ML RL model, or use instead second data previouslyoutput by the one or more third nodes 113 in one or more previous iterations of the training of the first ML RL model.

[0099] In some embodiments, the second MLM may be a Recurrent Neural Network (RNN). The training in this Action 302 may use the obtained respective first indication. In some examples, the training in this Action 302 may use the obtained respective first indication for every iteration. In other examples, the training in this Action 302 may use at least a subset of the obtained respective first indication for every iteration.

[0100] The first data may be output by training the first ML RL model with one or more third nodes 113 operating in the computer system 100.

[0101] In some examples, the state may be initialized with -1 values.

[0102] The training in this Action 302 may comprise, for every iteration, a) performing an action out of: i) “Action 0”: obtaining a respective set of first data from the one or more third nodes 113, which may be referred to herein as running a digital twin inference, and, ii) “Action 1”: obtaining a respective set of second data from one or more previous iterations, which may be referred to herein as keeping previous normalized synthetic counters in the state; b) determining a composite delayed error as an average of the squares of the relative difference between the synthetic counters and the delayed synthetic counters from all cells, or using the previous value for the synthetic counters without recomputing them, and c) determining a reward to the performed action based on whether the determined error is determined to be between two thresholds.

[0103] In some examples, in Action 301 , the first node 111 may obtain a maximum allowed deviation, that is, a percentage deviation between a next prediction and a previous value. The training in this Action 302 may be performed based on this maximum allowed deviation value.

[0104] The reward may be 1 when the decision may have turned out to be correct, and -1 when the decision may turn out to be incorrect. Action 0 may be correct when the composite delayed error may be outside the boundaries of the maximum allowed deviation. Action 1 may be correct when the composite delayed error may be within the boundaries of the maximum allowed deviation.

[0105] The first node 111 may give different penalties to incorrect 0 and 1 decisions based on their respective criticality.

[0106] In one possible implementation, the second MLM, e.g., RNNs, may accumulate the knowledge acquired from previous observations and predict the best next action at every iteration. The second MLM, e.g., RRN, may learn when it may need to run an inference of the one or more third nodes 113, e.g., the BM, thus refreshing all synthetic Key Performance Indicators (KPIs) to keep the estimation error of the synthetic KPI vs the real value of the KPI below a threshold from the real value.

[0107] The first node 111 may only perform the training in this Action 302 when the first node111 may obtain the respective set of first data from the one or more third nodes 113. That is, when the applied action may be “0”.

[0108] The frequency may be also referred to herein as “skip parameter” or “N”. The output of the training in this Action 302 of the second MLM may be understood to be the values of skip parameters N.

[0109] Action 303

[0110] In this Action 303, the first node 111 may determine that the second MLM is ready to be used for determining, as will be described in the next Action 304, a first frequency. The first frequency may be understood to be a frequency determined during an execution phase of the second MLM. The determining in this Action 303 may be based on a difference between using the first data and the second data being kept under a threshold, which may be referred to herein as a third threshold, over a certain number of iterations, which may be referred to herein as a second number of iterations, of the first plurality of iterations.

[0111] The difference between using the first data and the second data may be the composite delayed error.

[0112] Determining may be understood as calculating, estimating, deriving, or similar, or obtaining or receiving from another node. In some examples, the determining in this Action 303 may comprise after having trained the second MLM over the training dataset, all samples, validating the MLM with a validation dataset, or unseen data, and if the performance is good, e.g., low Mean Absolute Percentage Error, the first node 111 may deem the second MLM as suitable to use.

[0113] Action 304

[0114] In this Action 304, the first node 111 determines the first frequency, e.g., N, with which to select the first data output by training the first ML RL model with the one or more third nodes 113 operating in the computer system 100, to be used as input data to the second node 112 operating in the computer system 100, for training the first ML RL model, or use instead the second data previously output by the one or more third nodes 113 in the one or more previous iterations of the training of the first ML RL model. As stated earlier, each of the one or more third nodes 113 runs the respective behavioral model as the digital twin of the one or more variables of the communications system 101. The one or more variables are used to train the first ML RL model by the second node 112. The first ML RL model is to optimize a performance of the communications system 101.

[0115] Determining may be understood as calculating, estimating, deriving, or similar.

[0116] Expressed in other words the first data may be understood to be an output, e.g., a first output, of a behavioral model, e.g., a first behavioral model, of a digital twin, e.g., a first digitaltwin, that may result out of applying the output of the first ML RL model with the one or more third nodes 113 operating in the computer system 100.

[0117] The determining in this Action 304 may be performed by a determine_tolerance function that may be run by the first node 111.

[0118] The first frequency may be referred to herein as “N” or “skip parameter”.

[0119] The one or more previous iterations may be understood to refer to a certain reference iteration in which the determining in this Action 304 may be performed.

[0120] In some examples, the first node 111 may, in this Action 304, acquire the first frequency, that is, the N value, through a domain knowledge inspired heuristic algorithm.

[0121] In some embodiments, the determining in this Action 304 of the first frequency may be performed with the second MLM. In some of such embodiments, the determining in this Action 304 may be understood to be performed during an execution phase of the second MLM.

[0122] The first node 111 may be also referred to herein as a built-in predictor, e.g., a sequence model and Recurrent Neural Network (RNN), Long-shortterm memory (LSTM) associated at each third node of the one or more third nodes 113, e.g., each BM, to predict the first frequency, that is the skip parameter ‘N’.

[0123] One possible implementation for the determine_tolerance function run by the first node 111 may be based on the second MLM, e.g., RNNs, to accumulate the knowledge acquired from previous observations and predict the best next action at every iteration. The RRN may learn when it may need to run an inference of the one or more third nodes 113, e.g., the BM, thus refreshing all synthetic KPIs to keep the estimation error below a threshold from the real value.

[0124] Once the second MLM, e.g., the RRN, may be trained, the optimum first frequency, that is, the optimum N or skip parameter, may be found by averaging and / or dividing the number of proposed actions to keep the previous value divided by the number of proposed actions to run an inference of the one or more third nodes 113, e.g., the BM.

[0125] In some examples, the first node 111 may determine the first frequency by setting an average frequency as an average N value = 1+(number of actions=1 / number of actions=0). An alternative may be to use an average N value over a window of the last M iterations instead of the instantaneous action provided by the one or more third nodes 113. This may be understood to refer to the frequency in which the behavioral model may need to be run, the N parameter, before the training of the RNN may be completed, an initial value may need to be setup. A possibility may be to perform this with the instantaneous previous action, or to perform this in a window of M iterations, based on the number of actions=1 and the number actions=0, that is keep previous values vs. run an inference.

[0126] Is some embodiments, e.g., wherein the first node 111 may not have performed Action 302 and Action 303, the second MLM may have been obtained from the fourth node 114operating in the computer system 100. The fourth node 114 may have trained the second MLM. In such embodiments, one or more of the following may apply. According to a first option, the fourth node 114 and the one or more third nodes 113 may have respective levels of one or more of compute capability, latency and communication channel capability, that may differ beyond a respective first threshold. According to a second option, the fourth node 114 may be unable to yield the first data under a second time threshold. To yield the first data may be understood as to observe a selected aspect of the state of the communications system 101, this may be as simple as asking a gNB how many connections it may have, or something more complex as estimating path loss between a UE and a gNB. Both may take time due to different reasons.

[0127] In its simplest form, the first node 111, or skip parameter predictor, may start from an initial value of N defined by the operator, as original delay, and then increment N until the reward may start decreasing where in the case N may be reverted to its previous value. To avoid letting N reach a very high value, or even a low value, an upper and lower bound may be used. Is some embodiments, the determining in this Action 304 of the first frequency may be based on a first upper bound a first lower bound.

[0128] Alternatively, first node 111 may try to estimate the % of deviation as a function of N for each third node 113, e.g., BM, of the one or more third nodes 113. The first node 111 may be trained with deviation values correctly calculated at steps where the correct features may be available. At the same step we can measure the difference between the correct feature and the approximated one. Correct information may be understood here as a result of an observation that may be costly in time or compute, while an approximation may be understood to be less expensive and faster, but not as accurate. The approximation may also be obtained via compute or by simply using a previous value if there is negligible deviation.

[0129] Is some embodiments, the first frequency may be based on a respective first frequency determined for each of the one or more third nodes 113. In some of such embodiments, the determining in this Action 304 of the first frequency may comprise, for all of the one or more third nodes 113, to average first actions recommended by the second MLM to obtain as inferences from the one or more third nodes 113 and second actions recommended by the second MLM to keep as in the one or more previous iterations.

[0130] Action 305

[0131] In this Action 305, the first node 111 outputs the determined first frequency to the second node 112.

[0132] Outputting may be understood as providing, sending or similar. In some examples, the first node 111 may be the same or may be integrated with the first node 111. In other examples, the first node 111 and the second node 112 may be separate nodes.Embodiments of a computer-implemented method, performed by the second node 112, will now be described with reference to the flowchart depicted in Figure 4. The method is for handling input data to the first ML RL model. The second node 112 operates in the computer system 100.

[0133] Several embodiments are comprised herein. In some embodiments all the actions may be performed. In some embodiments, some of the actions may be optional. It should be noted that the examples herein are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description. The detailed description of some of the following corresponds to the same references provided above, in relation to the actions described for the first node 111, and will thus not be repeated here. For example, the second node 112 may be an Al optimizer, e.g., a DQN-based RL agent.

[0134] Action 400

[0135] In this Action 400, the second node 112 trains the first ML RL model with the one or more third nodes 113 operating in the computer system 100 for a second plurality of iterations.

[0136] The first ML RL model is to optimize the performance of the communications system 101.

[0137] In examples wherein the second node 112 may be a DQN-based RL agent, the training phase of the first ML RL model may start with preparatory steps related to the DQN algorithm where two neural networks, DQN, and target network may be initialized, and the experience buffer D may also be created. The DQN network may be used to train the agent, while the target network may be understood to be a copy of the DQN that may be used to stabilize the DQN and overcome catastrophic forgetting.

[0138] The remaining actions in the training phase may be performed in loop for K iterations, wherein every iteration may be referenced with i.

[0139] The training in this Action 400 comprises, for every iteration of the second plurality of iterations, performing Action 401 and Action 402. Any other details of the training of Action 401 may be understood to be performed according to known methods and will therefore not be described herein.

[0140] In some non-limiting examples, the method performed by the second node 112 may be implemented as an extension to a DDQN algorithm.

[0141] In some embodiments, the training in this Action 400 may further comprise, for everyiteration of the second plurality of iterations, one or more of Action 403-Action 406.

[0142] Action 401

[0143] In this Action 401 , the second node 112 obtains, from the first node 111 operating in the computer system 100, the first frequency with which to select the first data output by training the first ML RL model with the one or more third nodes 113 operating in the computer system 100, to be used as input data to the second node 112, for training the first ML RL model, or use instead the second data previously output by the one or more third nodes 113 in the one or more previous iterations of the training of the first ML RL model. As stated earlier, each of the one or more third nodes 113 runs the respective behavioral model as a digital twin of the one or more variables of the communications system 101.

[0144] The one or more variables are used to train the first ML RL model by the second node 112.

[0145] In some examples, the second node 112 may, in this Action 401 , acquire the first frequency, that is, the N value, through a domain knowledge inspired heuristic algorithm.

[0146] In some embodiments, the obtaining in this Action 401 of the first frequency may be performed with the second MLM run by the first node 111.

[0147] The second MLM may be the RNN.

[0148] One possible implementation for the determine_tolerance function run by the first node 111 may be based on the second MLM, e.g., RNNs, to accumulate the knowledge acquired from previous observations and predict the best next action at every iteration. The RRN may learn when it may need to run an inference of the one or more third nodes 113, e.g., the BM, thus refreshing all synthetic KPIs to keep the estimation error below a threshold from the real value.

[0149] Once the second MLM, e.g., the RRN, may be trained, the optimum first frequency, that is, the optimum N or skip parameter, may be found by averaging and / or dividing the number of proposed actions to keep the previous value divided by the number of proposed actions to run an inference of the one or more third nodes 113, e.g., the BM.

[0150] In some embodiments, the second MLM may have been trained by the fourth node 114 operating in the computer system 100. In some of such embodiments, one or more of the following may apply. According to one option, the fourth node 114 and the one or more third nodes 113 may have respective levels of one or more of compute capability, latency and communication channel capability, that may differ beyond the respective first threshold.

[0151] According to another option, the fourth node 114 may be unable to yield the first data under the second time threshold.In some embodiments, the first frequency may be based on a respective first frequency determined for each of the one or more third nodes 113, and the first frequency may be determined, for all of the one or more third nodes 113, to average first actions recommended by the second MLM to obtain as inferences from the one or more third nodes 113 and second actions recommended by the second MLM to keep as in the one or more previous iterations.

[0152] Action 402

[0153] In this Action 402, the second node 112 selects the one of the first data and the second data as input data for training the first ML RL model based on the obtained first frequency.

[0154] The selected input data may indicate a first state, “s”, of the communications system 101. The state may need to contain information regarding the tunable configuration parameters and the distribution of the normalized synthetic counters for a given configuration parameters value. Depending on that value the second node 112 either obtain data from the BM or the second node 112 may estimate that data for that BM.

[0155] Using this input, the second node 112 may construct state spaces partitioned per BM which the second node 112 may then use, in the next Action 403, to select the action that the second node 112 may need to perform.

[0156] Action 403

[0157] In this Action 403, the second node 112 may select, based on the selected input data, an action, “a”, to be performed on an environment of the communications system 101.

[0158] Action 404

[0159] In this Action 404, the second node 112 may perform the action, “a”, on the environment (env).

[0160] Action 405

[0161] In this Action 405, the second node 112 may obtain, based on the obtained first frequency, a second state, “s’”, of the environment resulting from the performance of the action, “a”. The second state may be obtained as third data output by training the first ML RL model with the one or more third nodes 113 operating in the computer system 100 or as fourth data previously output by the one or more third nodes 113 in one or more previous iterations of the training of the first ML RL model.

[0162] For every BM, depending on the previous decision, the second node 112 may either retrieve the new state of the BM or the second node 112 may rely on estimation. The second node 112 may include this information in s’ which may now be partitioned using the new orestimated observation per BM and the second node 112 may then include this information in the experience buffer D.

[0163] Action 406

[0164] In this Action 406, the second node 112 may determine a reward, “r”, corresponding to the action based on the obtained second state.

[0165] In some embodiments, the second node 112 may assign a respective weight to the input data. The respective weight may be larger for the first data than for the second data. The input data may be provided to the first ML RL model with the assigned respective weight.

[0166] Once the second node 112 may have acquired the first frequency, that is, the N value, either through a domain knowledge inspired heuristic algorithm or the data-driven approach described earlier in relation to Figure 3, the second node 112 may assign higher sample probabilities to the authentic state, action, state’, reward (s, a, s', r) experience tuples. This focus on authentic tuples may lead the second node 112, e.g., the DQN-based RL agent, to be more frequently trained with these high-priority experiences, ultimately resulting in enhanced performance with a reduced number of iterations. For instance, when the iteration step may be a multiple of N, the second node 112 may collect all relevant counters to compute state and reward, but the state obtained may be understood to be the "next state." Therefore, at step = nN, the second node 112 may encounter a tuple (s, A, R, S'), where the capitalized letters may be understood to signify authenticity, and at step = nN+1, the second node 112 may have a tuple (S, A, r, s'). Preserving both types of tuples may be recommended. For instance, when N=4, the second node 112 may maintain the data from iterations 0, 1, 4, 5, 8, 9, 12, 13... with a higher priority, or weight, in the experience replay buffer. Alternatively, the second node 112 may annotate data proportionately to their weight ranging from a high value to authentic data and a lower value to less authentic originating from non-important iterations.

[0167] In some embodiments, the method may further comprise one or more of Action 407 and Action 408.

[0168] Action 407

[0169] The second node 112 may train to minimize the distance between actual Q values and estimated long term reward. For every Y iteration, the second node 112 may update the target network which may be used to stabilize the training process.

[0170] In this Action 407, the second node 112 may determine that the first ML RL model is ready to be used to optimize the performance of the communications system 101, based on the determined reward exceeding a fourth threshold over a third number of iterations of the second plurality of iterations.Action 408

[0171] In this Action 408, the second node 112 may use the trained first ML RL model to optimize the performance of the communications system 101.

[0172] The execution phase may be similar to the training phase with the exception that the second node 112 may no longer maintain an experience buffer, nor the second node 112 may train the agent, nor the second node 112 may determine again the tolerance (how many iterations to wait) for retrieving vs. estimating data from a BM.

[0173] As stated earlier, in some non-limiting examples, the method performed by the second node 112 may be implemented as an extension to a DDQN algorithm. Overall the main difference between this the method performed by the second node 112 and the DDQN algorithm may be the generalization to multiple BMs to populate the state space, the use of a function to estimate how frequently, denoted as N, the second node 112 may need to retrieve data from the one or more third nodes 113, e.g., BM, using the first node 111, that is, the skip parameter predictor, as opposed to estimate the data, and the use of another function to update the skip parameter predictor related to the long term reward the second node 112 may have obtained. The process may be split into a training phase; where the second node 112 may be trained using input from multiple BMs, and either by asking input directly from the BM or estimating that input depending on the tolerance for each BM, and an execution phase where the second node 112 may be already trained, the delay tolerance for each BM may have been determined and, as such, the second node 112 may then either choose to retrieve data from the BM or estimate that data.

[0174] Figure 5 is a schematic sequence diagram illustrating a non-limiting example of the method performed by the second node 112 according to embodiments herein, which may be referred to as delay tolerant learning. Figure 5 particularly depicts a sequence diagram that describes the method performed by the second node 112 may work as an extension to the DDQN algorithm. Overall, the main difference between this sequence diagram and the DDQN algorithm may be understood to be the generalization to multiple third nodes 113, that is, BMs, to populate the state space, the use of a function to estimate how frequent, denoted as N, second node 112 may need to retrieve data from the BM, as determined by the skip parameter predictor, as opposed to estimate the data, step 4, according to Action 401, and step 18, and the use of another function to update the skip parameter predictor related to the long term reward we have obtained, step 12. The process may be split into a training phase, depicted in panel a), where the second node 112, that is, the agent / ai-based optimizer, may be trained using input from multiple BMs and either by asking input directly from the BM or estimating that input depending on the tolerance for each BM, and an execution phase, depicted in panel b), where the agent may be already trained, the delay tolerance for each BM may have been determinedand, as such, second node 112 may then either choose to retrieve data from the BM or estimate that data. Starting on the training phase on panel a), step 1, 2, 3 may be understood to be preparatory steps related to the DQN algorithm where two neural networks, a DQN, and a target network may be initialized, and the experience buffer D may be also created. The DQN network may be used to train the agent, while the target network may be a copy of the DQN used to stabilize the DQN and overcome catastrophic forgetting. The remaining steps in the training phase may be performed in loop for K iterations, wherein every iteration may be referenced with i. Within every i-th iteration, second node 112 may first ask the determine_tolerance function to indicate how frequently second node 112 may need to obtain new data for a given BM. One possible implementation for the determine_tolerance function run by the first node 111 may be based on the second MLM, e.g., RNNs, to accumulate the knowledge acquired from previous observations and predict the best next action at every iteration. The RRN may learn when it may need to run an inference of the one or more third nodes 113, e.g., the BM, thus refreshing all synthetic KPIs to keep the estimation error below a threshold from the real value. Once the second MLM, e.g., the RRN, may be trained, the optimum first frequency, that is, the optimum N or skip parameter, may be found by averaging and / or dividing the number of proposed actions to keep the previous value divided by the number of proposed actions to run an inference of the one or more third nodes 113, e.g., the BM. The state may contain information regarding the tunable configuration parameters and the distribution of the normalized synthetic counters for a given configuration parameters value. According to Action 402, depending on that value, second node 112 may either obtain data from the BM (step 5) or second node 112 may estimate that data for that BM (step 6). Using this input, the second node 112 may construct state spaces partitioned per BM which second node 112 may then use, according to Action 403, to select the action that the agent may need to perform (step 7). The action may be performed to the environment (env) 500 in step 8, according to Action 404. In step 8, and for every BM depending on the previous decision second node 112, according to Action 405, may either retrieve the new state of the BM or second node 112 may rely on estimation (step 9 and step 10). The second node 112 may include this information in s’ which may now be partitioned using the new or estimated observation per BM and second node 112 may then include this information in the experience buffer D. In Step 11, the second node 112 may calculate the reward for s, s’, a in accordance with Action 406. In Step 12, the second node 112 may update the tolerance for r, N. In Step 13, the second node 112 may append the tuple s’, s, a, r to the experience buffer D. Step 14-16 may be used to train the agent to minimize the distance between actual Q values and estimated long term reward. In Step 14, the second node 112 may select a random minibatch S <s,s’,a,r> from D. In Step 15, the second node 112 may calculate Q*(s_t, a_t <- r-t + y*Q_9(s_t+1, argmax(Q_0’(s_t+1,a’))). In Step 16, the second node 112 may calculate a gradient descent of Q*(s_ 0, a_t) - Q_0(s_t, a_t))A2. In step 17, for every Y iteration, secondnode 112 may update the target network (0’) which may be used to stabilize the training process. Continuing on the execution phase depicted in panel b), the execution phase may be similar to the training phase with the exception that second node 112 may no longer maintain an experience buffer, nor may the second node 112 train the agent to determine again the tolerance, that is, how many iterations to wait, for retrieving vs. estimating data from a BM. In its simplest form, the skip parameter predictor may start from an initial value of N defined by the operator, as original delay, and then increment N until the reward may start decreasing where in the case N may be reverted to its previous value. To avoid letting N reach a very high value, or even a low value, an upper and lower bound may be used. Alternatively, the first node 111 may try to estimate the % of deviation as a function of N for each BM. The predictor may be trained with deviation values correctly calculated at steps where the correct features may be available. At the same step the second node 112 may measure the difference between the correct feature and the approximated one. The execution phase may be performed in accordance with Action 408. In Step 18, the second node 112 may obtain the first frequency. In Step 19 and Step 20, the second node 112 may observe the state of the communications system 101 based on the obtained first frequency, obtaining either first data or second data. In Step 21, the second node 112 may select an action to be performed on the environment 500 according to a policy. In Step 22, the second node 112 may perform the selected action.

[0175] Figure 6 is schematic representation of the first node 111, that is, the predictor. Figure 6 illustrated the use of RRN as built-in predictor for the determine_tolerance function. The first node 111 may, based on an input state, perform an action, and obtain the reward. The state before and after performing the action may be obtained as first data or second data. The one or more third nodes 113, BMs, may receive as input a parameter change provided by the use case. The initial states may be filled in with -1 values. The state may be parameters configuration, normalized average and standard deviation of the synthetic counters. The action may be: 0: run a digital twin interface, or 1: keep previous normalized synthetic counters in the state. The reward may be 1 when the decision may be correct and -1 when the decision may be incorrect. Action 0 may be correct when the composite delayed error may be outside the boundaries of the maximum allowed deviation. Action 1 may be correct when the composite delayed error may be within the boundaries of the maximum allowed deviation. An average N value may be = 1+(number of actions=1 / number of actions=0). The composite delayed error may be the average of the squares of the relative difference between the synthetic counters and the delayed synthetic counters from all cells. The first node 111 may give different penalties to incorrect 0 and 1 decision based on the criticality of them. An alternative may be to use an average N value over a window of the last M iterations instead ofthe instantaneous action provided by the agent. The first ML RL model may be only trained when the applied action may be 0.

[0176] Figure 7 is a schematic representation of a layout of a network that was simulated with as simulator to perform an evaluation of a method according to embodiments herein. A study was carried out to evaluate the impact of a delay in the calculation of certain Key Performance Indicators (KPIs) used for optimization based on RL. This situation may happen when the KPIs may be computed by Distributed BMs, as in examples of embodiments herein. Some KPIs may be available at every optimization step, and some others with higher period N. The study was carried out using a modified version of an existing prototype for power optimization that may rely on the simulator and may emulate the behavior of the version that may rely on a DT. According to embodiments herein, not available KPIs may be replaced by the latest available value of the same KPIs. The experiment was defined as follows. The experiment was carried out with the next specifications. The simulated scenario comprised: a 2-ring scenario shown in Figure 7, with 19 sites with 3 cells per site. LTE users are depicted as black dots. The learning of the first ML RL model took place based on episodes. At each episode: i) only 21 cells at central site, depicted in solid black, as LTE macro cells, were to be optimized (green), and ii) an initialization randomly reset these parameters: Maxtx power: [37, 50] dBm, RET:

[0177] [0,14] deg, Frequency: {0.7, 0.8, 0.9, 1.8. 1.9, 2.1, 2.6} GHz, antenna height: [20, 30] m, inter site distance: [400, 1500] m, and offered traffic per cell: [1, 10] MHz. The simulated scenario also comprised that: a) an RL power optimization execution consisted in training an agent over 500 steps, b) scenarios were soft-reset, power values were assigned random values, after every period of max 20 steps, c) the exploration rate started at one with a decay factor of 0.99 per step, d) the experience replay was used with a buffer of samples contained in the last 50 steps, e) the state comprised a number of parameters and KPIs, where some of these KPIs were available every step, some of them were available every K steps, that is, iterations of the first ML RL model, and some others were built as the weighted average of KPIs at neighbor cells, where the weights were available once every K steps, f) the reward was the improvement of a utility function. This utility function was the weighted sum of five metrics, which represented power reduction, UL and DL traffic quality traffic, coverage traffic and congestion at the optimized cell and the neighbors. Some of these metrics were available only every N steps. The results of this assessment are depicted in Figure 8.

[0178] Figure 8 is a graphic representation of the results of the evaluation performed as described in Figure 8. Particularly, Figure 8 depicts the evolution of the reward improvement estimation error for different values of N: 1 , 4, 7 and 10 over the 500 steps. The evolution of the reward improvement estimation error was obtained for several values of N, as shown inFigure 8. This error was the absolute value of the difference between the accumulated reward calculated with current KPIs at the end of an episode and the same value calculated with delayed KPIs. It may be noticed how the error increases with the value of N. When selecting the value of N, there may therefore be understood to be a trade-off between the acceleration of the RL execution and the error in the estimation of the reward.

[0179] Figure 9 is a schematic representation of a non-limiting example of a computer system 100 according to embodiments herein. An ORAN implementation may serve as a cloud based implementation of embodiments herein. A Network Data Analytics Function (NWDAF) may be a good candidate for the first node 111, and / or the second node 112 as well with a new Model Training Logical Function (MTLF) handling the optimization of the network using input from multiple other MTLFs or different analytics reports that may be delayed thus suffering from the original issues recognized earlier herein, but also benefiting from the described methods. However, support for Reinforcement learning may need to be introduced for the embodiments herein to become more accessible to NWDAF. As to the O-RAN Implementation, embodiments herein may be implemented as rApps in the ORAN / Service Management and Orchestration (SMO) architecture. Depending on the implementation choices, the three technical components — the second node 112 (the Al optimizer), the one or more third nodes 113, (e.g., the RAN digital twin), and first node 111 (e.g., the skip parameter predictor) — may be grouped into a single rApp or executed as independent rApps, as illustrated in Figure 9. In the context of ORAN - the second node 112 (the Al optimizer) rApp may map to the Agent, illustrated in Figure 1, the RAN DT rApp 901 may map to the one or more third nodes 113, e.g., the BM, and the environment (env) may map to the Non real-time RIC 902, which may be used to collect information directly from cells. The RAN DT rApp 901 may comprise a dynamic DT 903 in accordance to embodiments herein, and a static DT 904. The RAN DT rApp 901 may receive CM parameter changes as input from the second node 112 and output a network state to the second node 112. The RAN DT rApp 901 may comprise UE level BMs 905 and Cell level BMs 906. The RAN DT rApp 901 may communicate with the non real-time RIC 902 via the R1 interface 907. In this way, BMs residing in the RAN DT rApp 901 may communicate with the environment if / when needed. Moreover, each BM within the RAN DT rApp 901 may contribute to the estimation of the state of the environment, or the future state of the environment given the output of the Al optimizer. However, this may be understood to be an expensive process as described herein, since execution of BMs may be computationally expensive. Therefore, this choice, whether the future network state is estimated via a BM or whether a previous value is considered may be determined by way of the first node 111 , the Skip Parameter (N) predictor rApp. In the previous setting, time may be considered as a constraint for this choice or, in other words, how much it may take to execute a BM. However,another constraint to consider may also be memory. If there is insufficient memory, a BM, as a process, may be offloaded and instead a previous value may be used. This may save memory and make room for other processes, such as other BMs, to use available memory.

[0180] As a summarized overview of the foregoing, embodiments herein may relate to the first node 111 , the “skip parameter predictor”, which may be understood to run the first ML RL model, which may be understood to be a model that may estimate the optimal frequency in which to retrieve data from multiple sources of a distributed BMs to feed to the second node 112, an Al-based optimizer, of a wireless cellular network. Moreover, this skip-parameter predictor may be integrated in a digital twin to categorize the retrieved data from the multiple sources of a distributed BMs and prioritize it to train the second ML RL model of the second node 112, the Al-based optimizer of the wireless cellular network.

[0181] Certain embodiments herein may provide one or more of the following technical advantage(s). By identifying how a slightly outdated and previously obtained measurement of a KPI may impact the choice of action and consequently the optimisation done in the communications system 101, the second node 112 may now choose to use previous values instead of interacting with BMs that may take a long time to produce their output. Moreover, the second node 112 may employ the same mechanism to determine which samples to include in the experience replay versus what to exclude to train the second node 112, using less data but with data of greater importance to the agent. The ability to overcome such delays may be understood to be particularly interesting when using external infrastructures such as certain web services, since they may come with a cost, even when they may not be fully used. More specifically, the resources for the optimizer may be allocated in a cloud infrastructure even if the optimizer process may be in an idle state. The embodiments herein may allow to reduce that cost, since different components may no longer be bottlenecked by missing input, and as such, may produce decisions faster, thus reducing the cost of using the external infrastructure.

[0182] Consequently, the frequency of the executions of the second node 112, the Al-based optimizer, may be increased, i.e. , speed, by skipping less relevant information from the distributed BMs when the accuracy of the new data may not justify the waiting time due to retrieval delay.

[0183] Embodiments herein may also improve the accuracy of the predictions of the first ML RL model, the Al-based model, thanks to giving higher importance, weights, to samples with less deviation.

[0184] One additional advantage may be that if the first node 111 , the skip parameter predictor, may determine that there may be no need for a given BM to estimate a given KPI,that may be offloaded from the cloud infrastructure, thus reducing memory / CPU resource consumption.

[0185] Figure 10 depicts an example of the arrangement that the first node 111 may comprise to perform the method described in Figure 3 and / or any of Figures 5-9. The first node 111 may be understood to be for handling the input data to the first ML RL model. The first node 111 is configured to operate in the computer system 100.

[0186] Several embodiments are comprised herein. It should be noted that the examples herein are not mutually exclusive. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description.

[0187] Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. The detailed description of some of the following corresponds to the same references provided above, in relation to the actions described for the first node 111, and will thus not be repeated here. For example, the second node 112 may be configured to be an Al optimizer, e.g., a DQN-based RL agent.

[0188] The first node 111 is configured and / or operable to, e.g. by means of a processing circuitry 1001 configured to, determine the first frequency with which to select the first data output by training the first ML RL model with the one or more third nodes 113 configured to operate in the computer system 100, to be used as input data to the second node 112 configured to operate in the computer system 100, for training the first ML RL model, or use instead the second data previously output by the one or more third nodes 113 in the one or more previous iterations of the training of the first ML RL model. Each of the one or more third nodes 113 is configured to run a respective behavioral model as a digital twin of the one or more variables of the communications system 101. The one or more variables are configured to be used to train the first ML RL model by the second node 112. The first ML RL model is configured to optimize the performance of the communications system 101.

[0189] The first node 111 is also configured and / or operable to, e.g. by means of the processing circuitry 1001 configured to, output the first frequency configured to be determined to the second node 112.

[0190] In some embodiments, the determining of the first frequency may be configured to be performed with the second MLM.

[0191] In some embodiments, the second MLM may be configured to be an RNN.

[0192] In some embodiments, the second MLM may be configured to have been obtained from the fourth node 114 configured to operate in the computer system 100. The fourth node 114 may be configured to have trained the second MLM. In some of such embodiments, one or more of the following may be configured to apply: i) the fourth node 114 and the one or morethird nodes 113 may be configured to have respective levels of one or more of compute capability, latency and communication channel capability, that may be configured to differ beyond the respective first threshold, and ii) the fourth node 114 may be configured to be unable to yield the first data under the second time threshold.

[0193] In some embodiments, the determining of the first frequency may be configured to be based on the first upper bound the first lower bound.

[0194] In some embodiments, the first node 111 may be further configured with the following three configurations.

[0195] In some embodiments, the first node 111 may be further configured and / or operable to, e.g. by means of the processing circuitry 1001 configured to, obtain, for every iteration of the first plurality of iterations, the respective first indication of the state of the communications system 101. The respective first indication may be configured to comprise the one or more respective values of the one or more parameters of the communications system 101. The one or more respective values may be configured to be outdated values configured to be obtained from the one or more third nodes 113 configured to operate in the computer system 100.

[0196] In some embodiments, the first node 111 may be further configured and / or operable to, e.g. by means of the processing circuitry 1001 configured to, train, for the first plurality of iterations, the second MLM to determine the frequency with which to select the first data from the one or more third nodes 113 to be used as input data to the second node 112 for training the first ML RL model, or use instead the second data previously output by the one or more third nodes 113 in the one or more previous iterations of the training of the first ML RL model. The training may be configured to use the respective first indication configured to be obtained.

[0197] In some embodiments, the first node 111 may be further configured and / or operable to, e.g. by means of the processing circuitry 1001 configured to, determine that the second MLM is ready to be used for determining the first frequency based on the difference between using the first data and the second data is kept under the third threshold over the second number of iterations of the first plurality of iterations.

[0198] In some embodiments, the first frequency may be configured to be based on the respective first frequency configured to be determined for each of the one or more third nodes 113. The determining of the first frequency may be configured to comprise, for all of the one or more third nodes 113, to average the first actions configured to be recommended by the second MLM to obtain as inferences from the one or more third nodes 113 and the second actions recommended by the second MLM to keep as in the one or more previous iterations.

[0199] The embodiments herein in the first node 111 may be implemented through one or more processors, such as a processing circuitry 1001 in the first node 111 depicted in Figure 10, together with computer program code for performing the functions and actions of the embodiments herein. A processor, as used herein, may be understood to be a hardwarecomponent. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the first node 111. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the first node 111.

[0200] The first node 111 may further comprise a memory 1002 comprising one or more memory units. The memory 1002 is arranged to be used to store obtained information, store data, configurations, schedulings, and applications etc. to perform the methods herein when being executed in the first node 111.

[0201] In some embodiments, the first node 111 may receive information from, e.g., the second node 112, the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100, through a receiving port 1003. In some embodiments, the receiving port 1003 may be, for example, connected to one or more antennas in first node 111. In other embodiments, the first node 111 may receive information from another structure in the computer system 100 through the receiving port 1003. Since the receiving port 1003 may be in communication with the processing circuitry 1001, the receiving port 1003 may then send the received information to the processing circuitry 1001. The receiving port 1003 may also be configured to receive other information.

[0202] The processing circuitry 1001 in the first node 111 may be further configured to transmit or send information to e.g., the second node 112, the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100, through a sending port 1004, which may be in communication with the processing circuitry 1001, and the memory 1002.

[0203] Those skilled in the art will also appreciate that the units comprised within the first node 111 described above as being configured to perform different actions, may refer to a combination of analog and digital circuits, and / or one or more processors configured with software and / or firmware, e.g., stored in memory, that, when executed by the one or more processors such as the processing circuitry 1001, perform as described above. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuit (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a System-on-a-Chip (SoC).

[0204] The first node 111 may be configured to perform any of the Actions described in relation to Figure 3 and / or any of Figures 5-9, e.g., by means of the processing circuitry 1001 within the first node 111, configured to perform any of such actions.Also, in some embodiments, different units comprised within the first node 111 may be configured to perform the different actions described above, implemented as one or more applications running on one or more processors such as the processing circuitry 1001.

[0205] Thus, the methods according to the embodiments described herein for the first node 111 may be respectively implemented by means of a computer program 1005 product, comprising instructions, i.e. , software code portions, which, when executed on at least one processing circuitry 1001, cause the at least one processing circuitry 1001 to carry out the actions described herein, as performed by the first node 111. The computer program 1005 product may be stored on a computer-readable storage medium 1006. The computer-readable storage medium 1006, having stored thereon the computer program 1005, may comprise instructions which, when executed on at least one processing circuitry 1001, cause the at least one processing circuitry 1001 to carry out the actions described herein, as performed by the first node 111. In some embodiments, the computer-readable storage medium 1006 may be a non-transitory computer-readable storage medium, such as a CD ROM disc, or a memory stick. In other embodiments, the computer program 1005 product may be stored on a carrier containing the computer program 1005 just described, wherein the carrier is one of an electronic signal, optical signal, radio signal, or the computer-readable storage medium 1006, as described above.

[0206] The first node 111 may comprise a communication interface configured to facilitate, or an interface unit to facilitate, communications between the first node 111 and other nodes or devices, e.g., the second node 112, the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100. The interface may, for example, include a transceiver configured to transmit and receive radio signals over an air interface in accordance with a suitable standard.

[0207] In other embodiments, the first node 111 may comprise a radio circuitry 1007, which may comprise e.g., the receiving port 1003 and the sending port 1004.

[0208] The radio circuitry 1007 may be configured to set up and maintain at least a wireless connection with the second node 112, the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100. Circuitry may be understood herein as a hardware component.

[0209] Hence, embodiments herein also relate to the first node 111 operative to operate in the computer system 100. The first node 111 may comprise the processing circuitry 1001 and the memory 1002, said memory 1002 containing instructions executable by said processing circuitry 1001, whereby the first node 111 is further operative to perform the actions described herein in relation to the first node 111, e.g., in Figure 3 and / or any of Figures 5-9.Figure 11 depicts an example of the arrangement that the second node 112 may comprise to perform the method described in Figure 4 and / or any of Figures 5-9. The second node 112 may be understood to be for handling the input data to the first ML RL model. The second node 112 is configured to operate in the computer system 100.

[0210] Several embodiments are comprised herein. It should be noted that the examples herein are not mutually exclusive. One or more embodiments may be combined, where applicable. All possible combinations are not described to simplify the description.

[0211] Components from one embodiment may be tacitly assumed to be present in another embodiment and it will be obvious to a person skilled in the art how those components may be used in the other exemplary embodiments. The detailed description of some of the following corresponds to the same references provided above, in relation to the actions described for the second node 112, and will thus not be repeated here. For example, the second node 112 may be configured to be an Al optimizer, e.g., a DQN-based RL agent.

[0212] The second node 112 is configured and / or operable to, e.g. by means of a processing circuitry 1101 configured to, train the first ML RL model with the one or more third nodes 113 configured to operate in the computer system 100 for the second plurality of iterations. The first ML RL model is configured to optimize the performance of the communications system 101. The training is configured to comprise, for every iteration of the second plurality of iterations: a) obtaining, from the first node 111 configured to operate in the computer system 100, the first frequency with which to select the first data output by training the first ML RL model with the one or more third nodes 113 configured to operate in the computer system 100, to be used as input data to the second node 112, for training the first ML RL model, or use instead the second data previously output by the one or more third nodes 113 in the one or more previous iterations of the training of the first ML RL model; Each of the one or more third nodes 113 is configured to run the respective behavioral model as the digital twin of the one or more variables of the communications system 101; the one or more variables are configured to be used to train the first ML RL model by the second node 112, and b) selecting one of the first data and the second data as input data for training the first ML RL model based on the first frequency configured to be obtained.

[0213] In some embodiments, the training may be further configured to comprise, for every iteration of the second plurality of iterations, one or more of the following four configurations.

[0214] In some embodiments, the training may be further configured to comprise selecting, based on the selected input data, the action to be performed on the environment of the communications system 101. The input data configured to be selected may be configured to indicate the first state of the communications system 101.

[0215] In some embodiments, the training may be further configured to comprise performing the action on the environment.In some embodiments, the training may be further configured to comprise obtaining, based on the first frequency configured to be obtained, the second state of the environment resulting from the performance of the action. The second state may be configured to be obtained as third data output by training the first ML RL model with the one or more third nodes 113 configured to operate in the computer system 100 or as fourth data previously output by the one or more third nodes 113 in one or more previous iterations of the training of the first ML RL model.

[0216] In some embodiments, the training may be further configured to comprise determining the reward corresponding to the action based on the second state configured to be obtained.

[0217] In some embodiments, the second node 112 may be further configured with one or more of the following two configurations.

[0218] In some embodiments, the second node 112 may be further configured and / or operable to, e.g. by means of the processing circuitry 1101 configured to, determine that the first ML RL model is ready to be used to optimize the performance of the communications system 101 , based on the reward configured to be determined exceeding the fourth threshold over the third number of iterations of the second plurality of iterations.

[0219] In some embodiments, the second node 112 may be further configured and / or operable to, e.g. by means of the processing circuitry 1101 configured to, use the trained first ML RL model to optimize the performance of the communications system 101.

[0220] In some embodiments, the second node may be configured to assign the respective weight to the input data. The respective weight may be configured to be larger for the first data than for the second data. The input data may be configured to be provided to the first ML RL model with the respective weight configured to be assigned.

[0221] In some embodiments, the obtaining of the first frequency may be configured to be performed with the second MLM configured to be run by the first node 111.

[0222] In some embodiments, the second MLM may be configured to be an RNN.

[0223] In some embodiments, the second may be is configured to have been trained by the fourth node 114 configured to operate in the computer system 100. In some of such embodiments, one or more of the following may apply: i) the fourth node 114 and the one or more third nodes 113 may be configured to have respective levels of one or more of compute capability, latency and communication channel capability, that may differ beyond the respective first threshold, and ii) the fourth node 114 may be configured to be unable to yield the first data under the second time threshold.

[0224] In some embodiments, the first frequency may be configured to be based on the respective first frequency configured to be determined for each of the one or more third nodes 113. In some of such embodiments, the first frequency may be configured to be determined, for all of the one or more third nodes 113, to average the first actions configured to berecommended by the second MLM to obtain as inferences from the one or more third nodes 113 and the second actions configured to be recommended by the second MLM to keep as in the one or more previous iterations.

[0225] The embodiments herein in the second node 112 may be implemented through one or more processors, such as a processing circuitry 1101 in the second node 112 depicted in Figure 11, together with computer program code for performing the functions and actions of the embodiments herein. A processor, as used herein, may be understood to be a hardware component. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the second node 112. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the second node 112.

[0226] The second node 112 may further comprise a memory 1102 comprising one or more memory units. The memory 1102 is arranged to be used to store obtained information, store data, configurations, schedulings, and applications etc. to perform the methods herein when being executed in the second node 112.

[0227] In some embodiments, the second node 112 may receive information from, e.g., the first node 111, the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100, through a receiving port 1103. In some embodiments, the receiving port 1103 may be, for example, connected to one or more antennas in second node 112. In other embodiments, the second node 112 may receive information from another structure in the computer system 100 through the receiving port 1103. Since the receiving port 1103 may be in communication with the processing circuitry 1101, the receiving port 1103 may then send the received information to the processing circuitry 1101. The receiving port 1103 may also be configured to receive other information.

[0228] The processing circuitry 1101 in the second node 112 may be further configured to transmit or send information to e.g., the first node 111, the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100, through a sending port 1104, which may be in communication with the processing circuitry 1101, and the memory 1102.

[0229] Those skilled in the art will also appreciate that the units comprised within the second node 112 described above as being configured to perform different actions, may refer to a combination of analog and digital circuits, and / or one or more processors configured with software and / or firmware, e.g., stored in memory, that, when executed by the one or more processors such as the processing circuitry 1101, perform as described above. One or more ofthese processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuit (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a System-on-a-Chip (SoC).

[0230] The second node 112 may be configured to perform any of the Actions described in relation to Figure 4 and / or any of Figures 5-9, e.g., by means of the processing circuitry 1101 within the second node 112, configured to perform any of such actions.

[0231] Also, in some embodiments, different units comprised within the second node 112 may be configured to perform the different actions described above, implemented as one or more applications running on one or more processors such as the processing circuitry 1101.

[0232] Thus, the methods according to the embodiments described herein for the second node 112 may be respectively implemented by means of a computer program 1105 product, comprising instructions, i.e. , software code portions, which, when executed on at least one processing circuitry 1101, cause the at least one processing circuitry 1101 to carry out the actions described herein, as performed by the second node 112. The computer program 1105 product may be stored on a computer-readable storage medium 1106. The computer-readable storage medium 1106, having stored thereon the computer program 1105, may comprise instructions which, when executed on at least one processing circuitry 1101, cause the at least one processing circuitry 1101 to carry out the actions described herein, as performed by the second node 112. In some embodiments, the computer-readable storage medium 1106 may be a non-transitory computer-readable storage medium, such as a CD ROM disc, ora memory stick. In other embodiments, the computer program 1105 product may be stored on a carrier containing the computer program 1105 just described, wherein the carrier is one of an electronic signal, optical signal, radio signal, or the computer-readable storage medium 1106, as described above.

[0233] The second node 112 may comprise a communication interface configured to facilitate, or an interface unit to facilitate, communications between the second node 112 and other nodes or devices, e.g., the first node 111, the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100. The interface may, for example, include a transceiver configured to transmit and receive radio signals over an air interface in accordance with a suitable standard.

[0234] In other embodiments, the second node 112 may comprise a radio circuitry 1107, which may comprise e.g., the receiving port 1103 and the sending port 1104.

[0235] The radio circuitry 1107 may be configured to set up and maintain at least a wireless connection with the first node 111 , the one or more third nodes 113, the fourth node 114, the radio network node 130, the device 140, and / or another structure in the computer system 100. Circuitry may be understood herein as a hardware component.Hence, embodiments herein also relate to the second node 112 operative to operate in the computer system 100. The second node 112 may comprise the processing circuitry 1101 and the memory 1102, said memory 1102 containing instructions executable by said processing circuitry 1101, whereby the second node 112 is further operative to perform the actions described herein in relation to the second node 112, e.g., in Figure 4 and / or any of Figures 5-9.

[0236] When using the word "comprise" or “comprising”, it shall be interpreted as non- limiting, i.e. , meaning "consist at least of".

[0237] The embodiments herein are not limited to the above-described preferred embodiments. Various alternatives, modifications and equivalents may be used. Therefore, the above embodiments should not be taken as limiting the scope of the invention.

[0238] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0239] As used herein, the expression “at least one of:” followed by a list of alternatives separated by commas, and wherein the last alternative is preceded by the “and” term, may be understood to mean that only one of the list of alternatives may apply, more than one of the list of alternatives may apply or all of the list of alternatives may apply. This expression may be understood to be equivalent to the expression “at least one of:” followed by a list of alternatives separated by commas, and wherein the last alternative is preceded by the “or” term.

[0240] Any of the terms processor and circuitry may be understood herein as a hardware component.

[0241] As used herein, the expression “in some embodiments” has been used to indicate that the features of the embodiment described may be combined with any other embodiment or example disclosed herein.

[0242] As used herein, the expression “in some examples” has been used to indicate that the features of the example described may be combined with any other embodiment or example disclosed herein.REFERENCES

[0243] 1. Ericsson RAN digital eBrief document, Wenfeng Hu, Yak Ng Molina, https: / / www.ericsson.com / en / network-automation / cognitive- technology?utm_medium=social_organic&utm_source=linkedin&utm_campaign=bcss_di gital-twins-ebrief_20230731#ebrief.

[0244] 2. https: / / pages.cs.wisc.edu / ~jphanna / papers / hanna2021importance.pdf.

[0245] 3. Schaul, Tom et al. “Prioritized Experience Replay.” CoRR abs / 1511.05952 (2015),

[0246] https: / / arxiv.org / abs / 1511.05952.

Claims

42CLAIMS:

1. A computer-implemented method, performed by a first node (111), the method being for handling input data to a first machine learning, ML, reinforcement learning, RL, model, the first node (111) operating in a computer system (100), the method comprising:- determining (304) a first frequency with which to select first data output by training the first ML RL model with one or more third nodes (113) operating in the computer system (100), to be used as input data to a second node (112) operating in the computer system (100), for training the first ML RL model, or use instead second data previously output by the one or more third nodes (113) in one or more previous iterations of the training of the first ML RL model, each of the one or more third nodes (113) running a respective behavioral model as a digital twin of one or more variables of the communications system (101), the one or more variables being used to train the first ML RL model by the second node (112), the first ML RL model being to optimize a performance of the communications system (101), and- outputting (305) the determined first frequency to the second node (112).

2. The method according to claim 1, wherein the determining (304) of the first frequency is performed with a second machine learning model, MLM.

3. The method according to claim 2, wherein the second MLM, is a Recurrent Neural Network.

4. The method according to any of claims 2-3, wherein the second MLM has been obtained from a fourth node (114) operating in the computer system (100), the fourth node (114) having trained the second machine learning model, and wherein one or more of:- the fourth node (114) and the one or more third nodes (113) have respective levels of one or more of compute capability, latency and communication channel capability, that differ beyond a respective first threshold, and - the fourth node (114) is unable to yield the first data under a second time threshold.

5. The method according to any of claims 1-4, wherein the determining (304) of the first frequency is based on a first upper bound a first lower bound.

436. The method according to any of claims 2-5, wherein the method further comprises:- obtaining (301), for every iteration of a first plurality of iterations, a respective first indication of a state of a communications system (101), the respective first indication comprising one or more respective values of one or more parameters of the communications system (101), the one or more respective values being outdated values obtained from the one or more third nodes (113) operating in the computer system (100),- training (302), for the first plurality of iterations, the second MLM to determine the frequency with which to select the first data from the one or more third nodes (113) to be used as input data to the second node (112) for training the first ML RL model, or use instead the second data previously output by the one or more third nodes (113) in the one or more previous iterations of the training of the first ML RL model, wherein the training (302) uses the obtained respective first indication, and- determining (303) that the second MLM is ready to be used for determining (304) the first frequency based on a difference between using the first data and the second data is kept under a third threshold over a second number of iterations of the first plurality of iterations.

7. The method according to any of claims 2-4 and any of claims 5-6, wherein the first frequency is based on a respective first frequency determined for each of the one or more third nodes (113), and wherein the determining (304) of the first frequency comprises, for all of the one or more third nodes (113), to average first actions recommended by the second MLM to obtain as inferences from the one or more third nodes (113) and second actions recommended by the second MLM to keep as in the one or more previous iterations.

8. A computer-implemented method, performed by a second node (112), the method being for handling input data to a first machine learning, ML, reinforcement learning, RL, model, the second node (112) operating in a computer system (100), the method comprising:- training (400) the first ML RL model with one or more third nodes (113) operating in the computer system (100) for a second plurality of iterations, the first ML RL model being to optimize a performance of the communications system (101), wherein the training (400) comprises, for every iteration of the second plurality of iterations:44- obtaining (401), from a first node (111) operating in the computer system (100), a first frequency with which to select first data output by training the first ML RL model with one or more third nodes (113) operating in the computer system (100), to be used as input data to the second node (112), for training the first ML RL model, or use instead second data previously output by the one or more third nodes (113) in one or more previous iterations of the training of the first ML RL model, each of the one or more third nodes (113) running a respective behavioral model as a digital twin of one or more variables of the communications system (101), the one or more variables being used to train the first ML RL model by the second node (112), and- selecting (402) one of the first data and the second data as input data for training the first ML RL model based on the obtained first frequency.

9. The method according to claim 8, wherein the training (400) further comprises, for every iteration of the second plurality of iterations, one or more of:- selecting (403), based on the selected input data, an action to be performed on an environment of the communications system (101), the selected input data indicating a first state of the communications system (101),- performing (404) the action on the environment,- obtaining (405), based on the obtained first frequency, a second state of the environment resulting from the performance of the action, wherein the second state is obtained as third data output by training the first ML RL model with the one or more third nodes (113) operating in the computer system (100) or as fourth data previously output by the one or more third nodes (113) in one or more previous iterations of the training of the first ML RL model, and - determining (406) a reward corresponding to the action based on the obtained second state, and wherein the method further comprises one or more of:- determining (407) that the first ML RL model is ready to be used to optimize the performance of the communications system (101), based on the determined reward exceeding a fourth threshold over a third number of iterations of the second plurality of iterations, and- using (408) the trained first ML RL model to optimize the performance of the communications system (101).

10. The method according to any of claims 8-9, wherein the second node (112) assigns a respective weight to the input data, and wherein the respective weight is larger for thefirst data than for the second data, and wherein the input data is provided to the first ML RL model with the assigned respective weight.

11. The method according to claim any of claims 8-10, wherein the obtaining (401) of the first frequency is performed with a second machine learning model, MLM, run by the first node (111).

12. The method according to claim 11, wherein the second MLM is a Recurrent Neural Network.

13. The method according to any of claims 11-12, wherein the second MLM has been trained by a fourth node (114) operating in the computer system (100), and wherein one or more of:- the fourth node (114) and the one or more third nodes (113) have respective levels of one or more of compute capability, latency and communication channel capability, that differ beyond a respective first threshold, and - the fourth node (114) is unable to yield the first data under a second time threshold.

14. The method according to any of claims 11-13, wherein the first frequency is based on a respective first frequency determined for each of the one or more third nodes (113), and wherein the first frequency is determined, for all of the one or more third nodes (113), to average first actions recommended by the second MLM to obtain as inferences from the one or more third nodes (113) and second actions recommended by the second MLM to keep as in the one or more previous iterations.

15. A first node (111), for handling input data to a first machine learning, ML, reinforcement learning, RL, model, the first node (111) being configured to operate in a computer system (100), the first node (111) being further configured to:- determine a first frequency with which to select first data output by training the first ML RL model with one or more third nodes (113) configured to operate in the computer system (100), to be used as input data to a second node (112) configured to operate in the computer system (100), for training the first ML RL model, or use instead second data previously output by the one or more third nodes (113) in one or more previous iterations of the training of the first ML RL model, each of the one or more third nodes (113) being configured to run a respective behavioral model as a digital twin of one or more variables of thecommunications system (101), the one or more variables being configured to be used to train the first ML RL model by the second node (112), the first ML RL model being configured to optimize a performance of the communications system (101), and- output the first frequency configured to be determined to the second node (112).

16. The first node (111) according to claim 15, wherein the determining of the first frequency is configured to be performed with a second machine learning model, MLM.

17. A second node (112), for handling input data to a first machine learning, ML, reinforcement learning, RL, model, the second node (112) being configured to operate in a computer system (100), the second node (112) being further configured to:- train the first ML RL model with one or more third nodes (113) configured to operate in the computer system (100) for a second plurality of iterations, the first ML RL model being configured to optimize a performance of the communications system (101), wherein the training is configured to comprise, for every iteration of the second plurality of iterations:- obtaining, from a first node (111) configured to operate in the computer system (100), a first frequency with which to select first data output by training the first ML RL model with one or more third nodes (113) configured to operate in the computer system (100), to be used as input data to the second node (112), for training the first ML RL model, or use instead second data previously output by the one or more third nodes (113) in one or more previous iterations of the training of the first ML RL model, each of the one or more third nodes (113) being configured to run a respective behavioral model as a digital twin of one or more variables of the communications system (101), the one or more variables being configured to be used to train the first ML RL model by the second node (112), and- selecting one of the first data and the second data as input data for training the first ML RL model based on the first frequency configured to be obtained.