Updating machine learning model for prediction of a metric in a network

Abstract

A computer-implemented method performed by a computing device for updating a ML model for prediction of a metric in a network. The method includes accessing (700) (i) data to be incorporated into the ML model that includes a set of data generators and a solver functionality, 5 and (ii) a similarity metric for determination of a similarity between the data and a respective data output from respective data generators. The method further includes determining (702) the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators; updating (704) at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data 0 generators; and providing (720) an updated set of data generators including the at least one of the respective data generator. Related methods and apparatus are also provided.

Description

P112470W001Updating Machine Learning Model For Prediction Of A Metric In A NetworkTECHNICAL FIELD

[0001] The present disclosure relates generally to computer-implemented methods performed by a computing device for updating a machine learning (ML) model for prediction of a metric in a network, and related methods and devices.BACKGROUND

[0002] A dependable network may be defined as a network where the delivery of required service performance for a given network service may be quantitatively ascertained according to an agreed service level objective.

[0003] With respect to such networks, it may be important to be able to determine what performance level the network or a network segment can guarantee at a given point in time. Predicting a metric corresponding to a performance level in the network / network segment may be done with a ML model.SUMMARY

[0004] There currently exist certain challenges. Some approaches for predicting a metric in a network may use a ML model that may degrade over time due to, e.g., a dynamic nature of the network in which additional, new data becomes available. The new data can impact the overall data distribution that the ML model has seen and, thus, the performance of the ML model. For example, new device types, new usage patterns, software upgrades, other environmental changes, etc. can cause data distributions of features and labels, upon which a ML model is built, to change; and the change can reduce ML model performance. Moreover, a particularly challenging aspect in predicting a metric with a ML model may be catastrophic forgetting when the ML model is updated with new insights over time. Another challenge may be the need for large amounts of data and memory, which may not be feasible in real world applications where memory may be limited and / or access to past data may be limited due to, e.g., privacy concerns. While generative replay may be an approach instead of saving raw data, such an approach uses one data generator. With use of one data generator, however, it may be challenging to give weights to previous tasks or to unlearn a task. The one data generator is learned to be agnostic to tasks; which limits knowledge that can be learned and may reduce the ability to remove selected previous knowledge form the ML model. Thus, it may be desirable to have a ML model that does not use, or minimizes use of,P112470W001a ML model that may degrade over time; mitigates a risk of catastrophic forgetting to ensure or improve a ML model’s reliability and robustness when deployed in networks; uses smaller amounts of data and / or memory; and / or uses more than one data generator to improve knowledge that can be learned and provide the ability to remove selected previous knowledge from the ML model.

[0005] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges.

[0006] Some embodiments provide a computer-implemented method performed by a computing device for updating a ML model for prediction of a metric in a network. The method comprises accessing (i) data to be incorporated into the ML model. The ML model comprises a set of data generators and a solver functionality, and (ii) a similarity metric for determination of a similarity between the data and a respective data output from respective data generators in the set of data generators. The method further comprises determining the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators; updating at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; and providing an updated set of data generators including the at least one of the respective data generator.

[0007] Other embodiments provide a computing device configured to update a ML model for prediction of a metric in a network. The computing device comprises the ML model comprising a set of data generators and a solver functionality; at least one processor; and at least one memory connected to the at least one processor and storing program code and / or the ML model that is executed by the at least one processor to perform operations. The operations comprise to access (i) data to be incorporated into the ML model. The ML model includes a set of data generators and a solver functionality, and (ii) a similarity metric for determination of a similarity between the data and a respective data output from respective data generators in the set of data generators. The operations further comprise to determine the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators; to update at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; and to provide an updated set of data generators including the at least one of the respective data generator.

[0008] Some embodiments include a computing device configured to update a ML model for prediction of a metric in a network. The computing device comprises the ML model comprising a solver functionality; at least one processor; and at least one memory connected to the at least one processor and storing program code and / or the ML model that is executed by the at least oneP112470W001processor to perform operations. The operations comprise to access (i) data to be incorporated into the ML model, and (ii) a similarity metric for determination of a similarity between the data and a respective data output from respective data generators in the set of data generators. The operations further comprise to determine the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators; to update at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; and to provide an updated set of data generators including the at least one of the respective data generator.

[0009] Other embodiments include a computer program product comprising a non-transitory storage medium including program code to be executed by at least one processor of a computing device configured to update a ML model for prediction of a metric in a network. The computer program product comprises program code and / or the ML model. Execution of the program code and / or the ML model causes the computing device to perform operations. The operations comprise to access (i) data to be incorporated into the ML model, and (ii) a similarity metric for determination of a similarity between the data and a respective data output from respective data generators in the set of data generators. The operations further comprise to determine the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators; to update at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; and to provide an updated set of data generators including the at least one of the respective data generator.

[0010] Certain embodiments may provide one or more of the following technical advantage(s). Based on inclusion of a set of data generators and a solver functionality, a ML model for predicting a metric in a network may be maintained / updated by amending the ML model with new data over time, while eliminating or minimizing the risk of catastrophic forgetting of previously learned knowledge. Moreover, in contrast to generative replay with one data generator, use of a set of data generators may make it easier to give weights to tasks or to unlearning tasks from the ML model. Further, if tasks have different data distributions, the set of data generators may have better performance in accuracy. Thus, inclusion of a set of data generators may allow for knowledge retention, while also allowing for purposeful forgetting. Moreover, a set of data generators may be maintained and their quality may improve over time as continual learning progresses. The individual data generators also may be used for other purposes and other use cases; or a composition of different data generators can form new data generators that can be used for other purposes and other use cases.P112470W001BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non- limiting embodiments of the present disclosure. In the drawings:

[0012] Figure 1 is a schematic drawing of an example of a dependable network with critical applications in need of service guarantees;

[0013] Figure 2 is a schematic drawing of an example of use of baseband statistics to train a ML model for one-way delay prediction per user equipment;

[0014] Figure 3 is a flowchart illustrating an example of operations of a computing device according to some embodiments;

[0015] Figure 4 is a diagram of a sequence of operations according to some embodiments;

[0016] Figures 5A and 5B are schematic drawings of testbed and a test area, respectively, of an example evaluation performed according to some embodiments;

[0017] Figure 6 is a block diagram of an example of an ML workflow in an open radio access network (ORAN) according to some embodiments;

[0018] Figure 7 is a flowchart illustrating operations of a computing device according to some embodiments;

[0019] Figure 8 is a block diagram of computing device according to some embodiments;

[0020] Figure 9 is a block diagram of a communication system in accordance with some embodiments;

[0021] Figure 10 is a block diagram of another communication system in accordance with some embodiments

[0022] Figure 11 is a block diagram of a wireless device in accordance with some embodiments;

[0023] Figure 12 is a block diagram of a network node in accordance with some embodiments; and

[0024] Figure 13 is a block diagram of a virtualization environment in accordance with some embodiments.DETAILED DESCRIPTION

[0025] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art, in which examples of embodiments ofP112470W001the present disclosure are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0026] An example of a dependable network is shown in Figure 1. The example includes a fifth generation (5G) / sixth generation (6G) system. The 5G / 6G system includes a user equipment (UE), a 5G / 6G access network, and a 5G / 6G core network. The 5G / 6G system further includes management, orchestration, and monetization functions; access, mobility, and network applications; cloud infrastructure and a data pipeline; and transport functions. In this example, critical applications have a need for a service guarantee(s) (e.g., a consistent minimum performance). Provision of the service guarantee(s) to the critical applications results in, among other things, value and monetization.

[0027] An approach for determining what performance level a network / network segment can guarantee at a point in time may be to use a predictive model trained using machine learning as illustrated in Figure 2. In the example in Figure 2, 5G baseband statistics including a buffer status, a signal-to-interference-plus-noise ratio (SINR), and scheduling data are used to train a ML model for one-way delay (OWD) prediction per user equipment (UE) as defined in e.g. ITU-T Y.1540.

[0028] Predicting a metric, such as OWD, may provide valuable insights for the network, UEs, and applications to address performance trends, deviations, and violations. For instance, the network can use delay predictions to enhance performance through explicit signaling in end-to-end communication protocols, for timely reconfiguration of network functions, and for service assurance.

[0029] ML models capable of such functionality may be integrated into a Third Generation Partnership Project (3 GPP) network data analytics function (NWDAF) system, for example, and may leverage observability data from a radio access network (RAN) to forecast UE delay. These predictions can then be communicated back to the RAN or to an application via a network exposure function (NEF) or an application function (AF).

[0030] Continual learning (CL) is a concept of learning a ML model for a number of tasks sequentially without forgetting knowledge obtained from the preceding tasks, where the data in the old tasks is no longer available during training new tasks.

[0031] A challenge of CL, however, is knowledge retention. It may be desirable for the ML model to remember old task knowledge after learning a new task, and to have good performanceP112470W001on both old tasks and new incoming tasks. Catastrophic forgetting is an outstanding challenge in CL. Catastrophic forgetting describes a problem where parameters, or semantic representations of parameters, learned for past tasks drift to the direction of new tasks during training. As in a deep neural network, due to gradient descent, neurons tend to forget past information when incorporating new information.

[0032] An approach to address the challenge, may be a replay -based approach. This approach may store samples in raw format or generate pseudo-samples with a generative ML model. Previous task samples are replayed while learning a new task to try to have less forgetting. The previous task samples may be either reused as ML model inputs for rehearsal, or to constrain optimization of the new task loss to prevent previous task interference.

[0033] CL with deep generative replay is a replay-based CL approach. Instead of replay memory for past data, in a deep generative replay framework, the ML model has one generator and may retain previously acquired knowledge by the concurrent replay of generated pseudo-data. In particular, a deep generative ML model generator may be trained to mimic past data. Generated data may then be paired with a corresponding response from the past task solver to represent old tasks. Such a ML model having a generator-solver pair may produce augmented data and desired target pairs as needed; and when presented with a new task, these produced pairs may be interleaved with new data to try to update the generator and solver network. Thus, in this approach, the ML model may both learn the new task without forgetting its own knowledge and teach other ML models with generated input- target pairs.

[0034] In such an approach, sequential training of such ML models may be performed with deep generative replay. Training a sequence of such ML models may be equivalent to continuous training of a single ML model while referring to its most recent copy. A new generator may be trained to mimic a mixed data distribution of real samples x and replayed inputs x' from a previous generator. A new solver may learn from real input-target pairs (x, y) and replayed input-target pairs (x', y'), where a replayed response y' is obtained by feeding generated inputs into previous solver.

[0035] There may be challenges with such CL approaches including, for example, data replay and generative replay. Data replay may be a strategy in CL if performance is the main objective. However, data replay may need large memory and often is not feasible in real world applications where access to past data is limited due to privacy -preserving, for example. Instead of saving raw data, generative replay may be an alternative strategy. However, in CL with one generator, it may be challenging to give weights to previous tasks or to unlearn tasks.P112470W001

[0036] In particular, in a CL approach of deep generative replay that uses one generator and one solver, the generator is learned to be agnostic to tasks. This marginalization may limit the knowledge that can be learned; and / or may reduce the ability to remove selected previous knowledge from the ML model.

[0037] Other types of CL techniques, such as regularization-based approaches and architectural-based approaches, also may have challenges for unlearning previous tasks in a structured manner.

[0038] In one approach, the performance of a ML-trained OWD model, utilizing data from a base station, may degrade over time due to the dynamic nature of the network. New UE types, new usage patterns, software upgrades, and other environmental changes can cause the data distributions of features and labels, upon which the ML model is built upon, to change. See e.g., Rao, A., Riaz, H., Zavodovski, A., Mochaourab, R., Berggren, V., & Johnsson, A., Generalizable one-way delay prediction models for heterogeneous UEs in 5g networks, NOMS 2024-2024 IEEE Network Operations and Management Symposium, (pp. 1-9), (2024 May). Such change can reduce the ML model performance. As discussed herein, a particularly challenging aspect in ML, and also for the task of predicting OWD, is catastrophic forgetting when a ML model is augmented with new insights over time. Mitigating this risk may be desirable to ensure or improve the ML model's reliability and robustness when deployed in networks.

[0039] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. Some examples include a method for updating (e.g., training and maintaining) a ML model based on a CL approach using deep generative replay with multiple generators to enhance ML model performance.

[0040] In one example, a computing device is provided for CL in the context of generative replay that may solve challenges related to OWD prediction in dynamic networks. Unlike CL that learns a single task-agnostic data generator, this example includes learning specialized data generators conditioned on the task. When a new task arrives, e.g. the network condition changes due to introduction of new UE types, its similarity to the previously seen tasks is assessed. If it is similar to one of the previous task groups, then it is assigned to the data generator of that group. If it is not similar to any of the previous task groups, then a new task group is constructed, and a new data generator is created and maintained. If the total number of data generators exceeds a defined upper-bound, the two most similar data generators are merged into one in this example.

[0041] In another example, a computer-implemented method performed by a computing device is provided for maintaining a ML model for predicting metrics in a telecommunication network, where the ML model includes a set of data generators [G] and a solver functionality S.P112470W001The method includes accessing / receiving new data D to be incorporated into the ML model, and a similarity metric M for determining similarity between D and outputs from generators in the set [G]. The method further includes determining pairwise similarity between D and each generator in [G]. In this example, if the similarity criterion is fulfilled for generator i, the generator i is updated with the data D. The method further includes providing the new set of generators to a requestor.

[0042] In yet another example, if the similarity criterion is not fulfilled, a new generator is created using the data D.

[0043] In another example, the method further includes determining pairwise similarity between each generator in the set [G] using the similarity metric M; and merging generators Gi and Gj if their similarity criterion is fulfilled.

[0044] In yet another example, the method further includes training the solver functionality S with data from the generators in new set Gnew.

[0045] Figure 3 is a flow chart of operations according to an example of the present disclosure. In operation 300, a computing device receives or accesses a new task Tnew, new data Dnew corresponding to the new task, a set of data generators [G], a solver S, a similarity metric, and an intent such as a maximum number of generators in [G].

[0046] In this example, Tnew is a learning objective that the ML model is trained to solve. Each task can have its own data distribution and set of labels. In one example for OWD prediction, a task can be defined in terms of configuration of baseband, set of UE types, movement patterns, etc.

[0047] Dnew may include both features and responses. In the OWD prediction example, Dnew corresponds to baseband features and OWD measurements.

[0048] A data generator may be referred to as G and may be based on a variational encoder (VAE) for tabular data (TVAE), for example, that has been trained on a task T with data D. The data generator is capable of providing samples without the need for the dataset D. Gi may be a specific data generator in the set of data generators [G]. In an OWD prediction example, the data generator Gi produces baseband feature values under the condition defined by a task T.

[0049] The solver S can be a ML model based on, e.g., a deep neural network that is trained using Tnew, Dnew, and the set of generators [G] in an effort to avoid catastrophic forgetting. In the OWD prediction example, the solver S is a ML model that predicts OWD from baseband data.

[0050] In operation 302, the computing device compares Dnew with each element in [G]. For example, a pairwise similarity between Dnew with each generator in [G] is determined.P112470W001

[0051] In operation 304, the computing device determines whether Dnew is similar to Gi. As shown in operations 306 and 308, respectively, if the similarity criterion if fulfilled (e.g., is above a threshold) for generator i, Gi is updated 306 with Dnew; and if the similarity criterion is not fulfilled (e.g., is below the threshold), a new generator Gnew is created / trained 308 using Dnew.

[0052] In operation 310, the computing device determines whether the number of data generators in the set of data generators [G] reached a maximum number.

[0053] In operation 312, if the number of the data generators reached the maximum, the computing device determines pairwise similarity between each generator in [G] using the similarity metric, and merge generators Gi and Gj if their similarity criterion is fulfilled (e.g., is above the threshold).

[0054] In operation 314, if the number of the data generators did not reach the maximum, the solver S is trained with data from the new set of data generators [G].

[0055] In one example, the similarity metric is a metric of a deviation between two probability distributions (e.g., probability distributions from real data and the data generated by the data generators). An example of such a metric is Kullback-Liebler (KL) divergence between Dnew and a generator G.

[0056] Figure 4 is a diagram of a sequence of signaling between a network device 400, a model manager 402, and a computing device 404 according to some examples.

[0057] In operation 406, network device 400 requests a prediction, such as a OWD prediction.

[0058] In operation 408, model manager 402 collects new data. If there is a change in network condition / data distribution, model manager 402 initiates the computing device 404, e.g. by providing access via an interface to the data.

[0059] In operation 410, in this example, the computing device 404 executes operations 302-314 of Figure 3.

[0060] In operation 412, the computing device 404 provides a status update to the model manager 402. For example, if a new data generator is created by the computing device 404, or if there is a change in the status of the data generators, it is communicated to the model manager 402.

[0061] In operation 414, the model manager 402 provides the prediction to network device 400, e.g., the OWD.

[0062] In another example, the signaling may be restricted to communication only between the network device 400 and the computing device 404. For example, this may happen when there are no data distribution updates identified by the network device 400 / model manager 402.

[0063] With reference to Figures 5 A and 5B, an evaluation of an example was performed using data from a 5G-mmWave indoor facility test area. In this example, baseband data wasP112470W001extracted, and OWD was measured between UEs and a network. The network includes a 5G mmWave (NSA antenna) data plane 502, a core network (NW) load server 504, long term evolution (LTE) (NSA anchor) control plane 506, a switch 508, a hub (repeater) 510, a measurement source 512, a measurement reflector 514, 5G modem 500a (load traffic), and 5G modem 500b (measurement traffic). For measurements of OWD, three different 5G UE types, referred to as UE 1, UE 2, and UE 3 were used. As shown in Figure 5B, the UEs could be positioned at stationary positions 1, 2, 3, or could be executed with various movement patterns including zigzag movement and / or rectangular movement within the test area as shown in Figure 5B (also referred to as IDs 8,0, or 7).

[0064] In the testbed shown in Figure 5A, measurements experiments, the various UE types, UE movement, and other configuration settings resulted in a distribution shift of OWD and features. To predict OWD, instead of training and maintaining an isolated ML model for each experiment, data of experiments with specific configuration settings could be treated as a sequence of tasks to a CL model.

[0065] The evaluation included a sequence of twelve (12) CL tasks with various UE types and UE movement as shown in the following Table 1:

[0066] As discussed in the example above, a CL task corresponds to new data Dnew that is previously unseen by the solver S.

[0067] The evaluation includes four approaches: a naive method, a cumulative method, a deep generative replay approach, and a multi generator example in accordance with the present disclosure.

[0068] In the naive method, a lower bound baseline algorithm trains a ML model with data of each task. That is, the solver is finetuned using data from the new task, without access to previous task data.

[0069] In the cumulative method, an upper bound baseline algorithm was used. For each task, the cumulative method accumulated all data from previous tasks and data of a current task, and retrained the solver from scratch. It is noted that the cumulative method presents challenges and may not be feasible in real world applications, as it needs access to all the previous data.P112470W001

[0070] In the deep generative replay approach, a TVAE was used as a data generator in deep generative replay.

[0071] In the multi generator example, based on domain knowledge of UE type, the 12 tasks above were divided to two groups. Groupl included all tasks of UE2, e.g. task 1, task 3, task 5, task 7, task 9, task 11. Group2 included all tasks of UE1, e.g. task 2, task 4, task 6, task 8, task 10, task 12. For each group, there was one data generator which was trained with a dataset of tasks in the group, and could generate samples for the tasks in the group. The TVAE was used in the multigenerator.

[0072] Further in this evaluation, as OWD prediction is a regression problem, the ML model performance was evaluated with a standard percentage error metric - mean absolute percentage error (MAPE). Lower values of MAPE indicate a better performing ML model.

[0073] Average accuracy (ACC) and backward transfer (BWT) are two examples of performance evaluation metrics of a CL model. ACC typically evaluates overall performance of the tasks learned so far. In the evaluation, after training on all tasks, average MAPE over all tasks’ test set was evaluated with respect to ACC. A lower value of ACC indicates better accuracy. BWT evaluates the memory stability of old tasks. In the evaluation, BWT was evaluated with MAPE on task i after training on last task, minus MAPE on task i after training on task i, average over i=l ,...n-1, n is number of tasks. A lower value of BWT indicates a better BWT.

[0074] The following Table 2 shows results of the metrics for the four approaches in the example evaluation:

[0075] As shown in Table 2, the multi-generator method shows lower scores of metrics than the deep generative replay and naive approaches. For ACC metrics, the gain of the multi-generator method is 15% compared with deep generative replay approach, which indicates less catastrophic forgetting.

[0076] A CL method of some embodiments may ensure or improve maintaining a ML model over time. The method may reduce requirements related to data storage and on ML model training. Instead of retraining with all available data, as in the cumulative approach for example, the MLP112470W001model is finetuned and updated. This may reduce energy consumption. Moreover, the specific multi-generator method may have much less storage overhead compared with saving raw data as in the cumulative approach, for example.

[0077] Figure 6 is a block diagram of an example of a ML workflow in an ORAN architecture in accordance with some embodiments. Figure 6 illustrates an example of an ORAN architecture that includes ORAN infrastructure 620 including ORAN nodes 622a-n; an ORAN node 616 performing as an inference host; a ML model management apparatus 606; a continuous operations apparatus 614; and interfaces A1 / O1 / E2. Figure 6 also includes an overview of operations that include data collection 600, data preparation 602, ML training 604, validation 608, publication 610, deployment 612, monitoring 614a, verification 614b, analysis 614c, continuous optimization 614d, ML model execution 618 and inference, and performance feedback. Operations of some embodiments herein are executed as part of a ML workflow component within the service management and orchestration (SMO) framework. In particular, operations of the method of some embodiments can run as part of the operations ML training 604 and / or validation 608 in Figure 6.

[0078] In some examples, an indicator to the computing device that the ML model needs to be retrained originates from the continuous operations apparatus in Figure 6. The indicator can be based on data analysis, or signals from other sources.

[0079] Figure 7 is a flowchart illustrating operations of a computing device according to some embodiments. Referring to Figure 7, some embodiments are directed to a computer-implemented method performed by a computing device for updating a ML model for prediction of a metric in a network. The method includes accessing 700 data to be incorporated into the ML model. The ML model includes a set of data generators and a solver functionality; and accessing 700 a similarity metric for determination of a similarity between the data and a respective data output from respective data generators in the set of data generators. The method further includes determining 702 the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators; and updating 704 at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators. The method further includes providing 720 an updated set of data generators including the at least one of the respective data generator.

[0080] In some embodiments, the data corresponds to a new task when there is at least one of a change in a network condition and a change in a data distribution.

[0081] Accessing 700, in some embodiments, further includes accessing an intent that includes a maximum number of data generators for the set of data generators. In someP112470W001embodiments, the intent can change over time. Thus for example, the maximum number of data generators can change.

[0082] The similarity metric, in some embodiments, is a metric that measures deviation between the data and the data output from a respective data generator.

[0083] In some embodiments, the updated set of data generators is provided for the solver functionality to predict the metric.

[0084] As shown in operation 706, in some embodiments, the method further includes creating a new data generator with the data when the similarity criterion is not met for at least one of the respective data generators.

[0085] Some embodiments further include training 708 the ML model including the new data generator with the data; checking 710 whether a number of the data generators in the ML model exceeds the maximum number of data generators; when the number exceeds the maximum number of data generators, determining 712 whether the similarity criterion is met, based on the similarity metric, between the data and the respective data output from the respective data generators; and merging 714 two respective data generators in the set of data generators, which results in a reduced number of data generators in the set of generators, when the similarity criterion for each of the two respective data generators is met.

[0086] Merging 714, in some embodiments, includes merging two or more respective data generators that are most similar.

[0087] In some embodiments, the method further includes training 716 the solver functionality with the data from the updated set of data generators or a subset of the data generators. Training 716 the solver functionality with data from a subset of the data generators may be performed, for example, to remove contribution of one of the data generators from the ML model for reasons such as being biased, erroneous, etc. Training 716 may be performed in a variety of manners including, without limitation, using equal amounts of data from each data generator, or using more data from an important data generator and less data from another data generator. In another example, if there is newly collected data, the newly collected data can be incorporated together with the data generated from the data generators into one set of data.

[0088] The method further includes, in some embodiments, removing 718, from the set of data generators, a data generator that has learned from undesirable data such as data that degraded over time.

[0089] In some embodiments, the method further includes predicting 722 the metric with the ML model including the updated set of data generators and the solver functionality; and providing 724 the predicted metric towards a device that requested the metric.P112470W001

[0090] In some examples, the similarity criterion is a threshold; if the similarity is above the threshold for data generator i, Gi is updated with Dnew; and if the similarity is below the threshold, a new data generator Gnew is created / trained using Dnew.

[0091] While examples of the present disclosure are discussed in the non-limiting context of a network for predicting a metric, the network can be a telecommunications network for example.

[0092] In one example, a ML model is maintained for predicting a metric in a telecommunication network. The ML model includes a set of data generators G and a solver functionality S. The method includes receiving new data D to be incorporated into the ML model, and a similarity metric M for determining similarity between D and outputs from generators in the set G. The method further includes determining pairwise similarity between D and each generator in the set of generators G. In this example, if the similarity is above a threshold for generator i, generator i is updated with the data D; and the new set of generators is provided to a requester of the metric.

[0093] In some embodiments, prediction of the metric in the network includes prediction of at least one of a OWD, a round-trip time (RTT), a packet loss, a corrupted packet, a reference signal received power (RSRP), and a bit error rate (BER) in a telecommunications network.

[0094] A technical advantage of some embodiments may include maintaining an OWD prediction model by amending the model with new data over time, without running the risk of catastrophic forgetting of previously learned knowledge.

[0095] Compared with generative replay with one data generator, a technical advantage of multiple data generators may be that it is easier to give weights to tasks or unlearning tasks from the ML model. Additionally or alternatively, if tasks have different data distributions, use of multiple data generators may result in better performance in accuracy.

[0096] While embodiments of the present disclosure allow for knowledge retention, in contrast to other approaches such as generative replay with one data generator, another technical advantage based on use of multiple data generators may include purposeful forgetting. Purposeful forgetting may be enabled by removing a target data generator that has been learned using undesirable data, thus enabling a form of unlearning or model repair.

[0097] Yet a further technical advantage of the inclusion of multiple data generators may include that the data generators may be maintained and their quality may improve over time as CL progresses. Moreover, the individual data generators can be used for other purposes and other use cases. Alternatively or additionally, a composition of different data generators can form new data generators that can be used for other purposes and other use cases.P112470W001

[0098] Further, the CL learning method of some embodiments may provide a technical advantage of maintaining a ML model over time. The method may reduce requirements related to data storage, and requirements on ML model training. For example, instead of retraining with all available data, the ML model may be finetuned and updated. This may reduce energy consumption. Additionally or alternatively, use of multiple data generators may use much less storage overhead compared with saving raw data.

[0099] The computing device can be any apparatus and / or virtual entity in a network configured to perform operations discussed herein. Examples of a computing device include, without limitation, a computer, a wireless device, a network node, a distributed node, a cloudbased node, and / or an O-RAN node.

[0100] Operations of a computing device can be performed by the computing device 800 of Figure 8. Operations of the computing device (implemented using the structure of Figure 8) have been discussed with reference to the flow chart of Figure 7 according to some embodiments of the present disclosure. Operations 706-718 and 722-724 of Figure 7 may be optional with respect to some embodiments of computing devices and related methods. Modules may be stored in memory 804 and / or the generators [G] 810, solver 814 of Figure 8, for example. These modules may provide instructions so that when the instructions of a module are executed by respective computing device processor 802 (also referred to herein as processing circuitry), computing device 800 performs respective operations of the flow chart of Figure 7.

[0101] As shown in Figure 8, the computing device 800 includes processor 802 that is operatively coupled to memory 804, network interface 806, generators [G] 810, solver 814, storage 812 for data and meta data 816, and / or any other component, or any combination thereof. Certain computing devices may utilize all or a subset of the components shown in Figure 8. Moreover, in some embodiments, components of the computing device may be distributed including, e.g., storage 810 for generators [G]; storage 812 for data; program code 808 (which may include program code for performing the operations of Figure 7); and meta data component 816. The level of integration between the components may vary from one computing device to another computing device. Further, certain computing devices may contain multiple instances of a component, such as multiple processors, memories, etc. Program code 808 may include code that runs inside computing device 800 that is responsible for updating / maintaining the ML model comprising the set of data generators and the solver functionality. In an example, such program code 808 also is responsible for providing predictions, e.g. OWD predictions, towards a requesting entity in the network, or an application in a UE for example.P112470W001

[0102] The processor 802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 804 and / or generators [G] 810, solver 814. The processor 802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processor 802 may include multiple central processing units (CPUs).

[0103] In the example, the network interface 806 includes an interface to / towards a requesting device for providing new data to the computing device 800, requesting predictions from the solver 814, updating meta data component 816, etc. Moreover, network interface 806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a display, a monitor, a printer, another output device, or any combination thereof. An input device may allow a user to capture information into the computing device 800. Examples of an input device include a touch-sensitive or presencesensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, a force sensor, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0104] Storage 810 is included for data generators [G], e.g., for generators Gi, . . . Gnconnected to a specific solver 814.

[0105] Storage 812 is included for incoming data / tasks, e.g. new dataset Dnew,. The data arrives via the network interface 806.

[0106] The ML model includes solver 814. The ML model may be trained with the objective of avoiding catastrophic forgetting. Such training is done using data from the data generators, in addition to available data from the network.

[0107] Meta data component 816 includes metrics, such as the similarity metric or limits on the number of data generators allowed. The metric(s) may be configured via the network interface 806. An example of a similarity metric is the KL-divergence between new data Dnew and generated data obtained from a generator Gi.P112470W001

[0108] The memory 804 and / or generators [G] 810, solver 814 may be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable readonly memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 804 and / or generators [G] 810, solver 814 includes one or more application programs in program code 808, such as the ML model, an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data. The memory 804 and / or generators [G] 810, solver 814 may store, for use by the computing device 800, any of a variety of various operating systems or combinations of operating systems.

[0109] The memory 804 and / or generators [G] 810, solver 814 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 804 and / or generators [G] 810, solver 814 may allow the computing device 800 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 804 and / or generators [G] 810, solver 814 which may be or comprise a device-readable storage medium.

[0110] The processor 802 may be configured to communicate with a network using the network interface 806. The network interface 806 may comprise one or more communication subsystems. The network interface 806 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another computing device, local computing device, edge node, cloud node, etc.). Each transceiver may include a transmitter and / or a receiver appropriate to provide network communications (e.g., optical, electrical, and so forth).P112470W001

[0111] In the illustrated embodiment, communication functions of the network interface 806 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / intemet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0112] Although the computing device described herein may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the computing device, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processor and the network interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0113] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discreteP112470W001device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0114] In certain embodiments, a computing device (404, 604, 608, 800, 908, 912, 1012, 1032, 1100, 1200, 1302) configured to update a ML model (808, 810, 814) for prediction of a metric in a network is provided. The computing device includes a ML model that includes a set of data generators (810) and a solver functionality (814); at least one processor (802); and at least one memory (804) connected to the at least one processor (802) and storing program code (808) that is executed by the at least one processor to perform operations. The operations include to perform some or all of the functionality described herein.

[0115] In certain embodiments, a computer program product is provided including a non-transitory storage medium (804) including program code (808) to be executed by at least one processor (802) of a computing device (404, 604, 608, 800, 908, 912, 1012, 1032, 1100, 1200, 1302) configured to update a ML model (808, 810, 814) for prediction of a metric in a network. The computing device includes program code (808) and / or a ML model (808, 810, 814). Execution of the program code and / or ML model causes the computing device to perform operations. The operations include to perform some or all of the functionality described herein.

[0116] Figure 9 shows an example of a communication system 900 in accordance with some embodiments.

[0117] In the example, the communication system 900 includes a telecommunications network 902 that includes an access network 904, such as a RAN, and a core network 906, which includes one or more core network nodes 908. The access network 904 includes one or more access network nodes or base stations of various types, access network nodes 910 A and 910B are depicted (which may be collectively referred to as network nodes 910), or any other similar 3GPP access nodes or non-3GPP access points (APs). Some embodiments of the access network 904 may include more than one access network technology. The network nodes 910 of access network 904 facilitate direct or indirect connection of wireless devices, also referred to as UEs, such as by connecting UEs 912A, 912B, 912C, and 912D (one or more of which may be generally referred to as UEs 912) to the core network 906 over one or more wireless connections.

[0118] Moreover, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it willP112470W001be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunications network 902 includes one or more ORAN network nodes. An ORAN network node is a network node in the telecommunications network 902 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other network nodes to implement one or more functionalities of any network node in the telecommunications network 902, including one or more access network nodes 910 and / or core network nodes 908.

[0119] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU-CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). An ORAN network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an Al, Fl, Wl, El, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN network node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an O-2 interface defined by the O-RAN Alliance or comparable technologies.

[0120] The network nodes 910 facilitate direct or indirect connection of one or more UEs 912 to the core network 906 over one or more wireless connections. Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 900 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 900 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0121] The UEs 912 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with theP112470W001network nodes 910 and other communication devices. Similarly, the network nodes 908, 910 are arranged, capable, configured, and / or operable to communicate directly or indirectly (e.g., via other devices of telecommunications network 902) with the UEs 912 and / or with other network nodes or equipment in the telecommunications network 902 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunications network 902. More specifically, UEs 912 may send messages, data, and / or other signals to network nodes 908, 910 or other elements of the telecommunications network 902 by transmitting such signals to the relevant device directly without the signals passing through any intervening devices or by transmitting such signals to the relevant device indirectly through an intervening device (or multiple intervening devices) that then transmit the signal to the relevant device. Similarly, network nodes 908, 910 may send messages, data, and other signals to UEs 9122, other network nodes 908, 910, and other devices in telecommunications network 902 directly or indirectly. As one specific example, a core network node 108 may transmit a particular message to a UE 912 by transmitting the message to an access network node 910 that will then transmit the message to the intended UE 912. Similarly, a core network node 108 may receive a particular message from a UE 912 by receiving the message from an access network node 910 that itself received the message from the UE 912.

[0122] In the depicted example, the core network 906 connects elements of the access network 904 (e.g., one or more of the network nodes 910) to one or more host computing systems, such as host 916. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 906 includes one or more core network nodes (e.g., core network node 908) of various types, one or more of which may be generally referred to as network nodes 908. Network nodes 908 are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, access network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 908. Example core network nodes provide functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDE), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0123] The host 916 may be under the ownership or control of a service provider other than an operator or provider of the access network 904 and / or the telecommunications network 902.P112470W001The host 916 may be operated by the service provider or on behalf of the service provider. The host 916 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0124] As a whole, the communication system 900 of Figure 9 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system 900 may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); LTE, and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (Wi-Fi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (Wi-Max), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, Li-Fi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox. Moreover, the communication system 900 may be configured to support multiple different standards, protocols, or other rule sets, with individual components supporting all of the relevant rule sets or with different components or sub-systems within the communication system 900 supporting different standards, protocols, or rule sets.

[0125] As one example, in certain embodiments, access network 904 may contain some access network nodes 910 that support 3GPP radio access technologies (RAT), such as LTE or NR, while other access network nodes 910 support (or the same access network nodes 910 additionally support) non-3GPP RATs, such as Wi-Fi or a proprietary RAT. As another example, telecommunications network 902 may support multiple generations of related communication standards (e.g., 4G and 5G 3GPP communication standards) and, as a result, may include an access network 904 and / or a core network 906 that supports multiple different standard generations or may include multiple access networks 904 and / or multiple core networks 906 with individual networks 904, 906 supporting different standard generations.

[0126] Telecommunications network 902 may support network slicing to provide different logical networks to different devices that are connected to the telecommunications network 902. For example, the telecommunications network 902 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile BroadbandP112470W001(eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0127] In some examples, one or more of the UEs 912 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 904 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 904. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR- DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0128] In the example, the hub 914 communicates with the access network 904 to facilitate indirect communication between one or more UEs (e.g., UE 912C and / or 912D) and network nodes (e.g., network node 910B). In some examples, the hub 914 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 914 may be a broadband router enabling access to the core network 906 for the UEs. As another example, the hub 914 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 910, or by executable code, script, process, or other instructions in the hub 914.

[0129] As another example, the hub 914 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 914 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 914 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 914 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 914 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0130] The hub 914 may have a constant / persistent or intermittent connection to the network node 910B. The hub 914 may also allow for a different communication scheme and / or schedule between the hub 914 and UEs (e.g., UE 912C and / or 912D), and between the hub 914 and the core network 906. In other examples, the hub 914 is connected to the core network 906 and / or one or more UEs via a wired connection. Moreover, the hub 914 may be configured to connect to an M2M service provider over the access network 904 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 910 while stillP112470W001connected via the hub 914 via a wired or wireless connection. In some embodiments, the hub 914 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 91 OB. In other embodiments, the hub 914 may be a nondedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 910B, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0131] Figure 10 is another example of a communication system 1000 according to some embodiments. As used herein, the communication system 1000 includes multiple access points (APs) 1010 (with four exemplary APs 1010A, 1010B, 1010C, and 1010D being depicted) and multiple wireless devices, referred to in the context of communication system 1000 as stations (STAs) 1012 (referred to individually as STA 1012A, STA 1012B, STA 1012C, STA 1012D, and STA 1012E). STA 1012A is served by AP 1010A in a first basic service set (BSS) 1020A. STA 1010B and STA 1010C are served by AP 1010B in a second BSS, BSS 1020B. STA 1012D is served by AP 1010C in a third BSS, BSS 1020C. STA 1012E is served by AP 1010D in a fourth BSS, BSS 1020D. Stations 1012 may be non-AP STAs and correspond to various kinds of wireless devices, for example, user terminals, such as mobile or stationary computing devices like smartphones, laptop computers, desktop computers, tablet computers, gaming devices, headmounted displays (HMDs) for Augmented Reality (AR) or Virtual Reality (VR), or the like. Further, stations 1012 could, for example, correspond to other kinds of equipment like smart home devices, printers, multimedia devices, data storage devices, or the like.

[0132] Each of STAs 1012 may connect through a radio link to one of APs 1010. For example, depending on location or channel conditions experienced by a given STA 1012, the STA may select an appropriate AP and BSS for establishing the radio link. The radio link may be based on one or more orthogonal frequency-division multiplexing (OFDM) carriers from a frequency spectrum that is shared on the basis of a contention-based mechanism, e.g., an unlicensed or license exempt band like 2.4 GHz Industrial, Scientific, and Medical (ISM) band, the 5 GHz band, the 6 GHz band, or the 60 GHz band.

[0133] Each AP 1010 may provide data connectivity to STAs 1012 connected to a particular AP 1010. As illustrated, APs 1010 may be connected to a data network 1030. In this way, APs 1010 may also provide data connectivity between STAs 1012 and other entities, e.g., to one or more servers, service providers, data sources, data sinks, user terminals, or the like. Accordingly, the radio link established between a given STA 1012 and its serving AP 1010 may be used for providing various kinds of services to STA 1012, e.g., a voice service, a multimedia service, or other data service. Such services may be based on applications that are executed on STA 1012P112470W001and / or on a device linked to STA 1012. By way of example, Figure 10 illustrates an application service platform 1032 provided in data network 1030. The application(s) executed on STA 1012 and / or on one or more other devices linked to STA 1012 may use the radio link for data communication with one or more other STA 1012 and / or the application service platform 1032, thereby enabling utilization of the corresponding service(s) at STA 1012.

[0134] Figure 11 shows a wireless device 1100, which may be configured to operate in communication system 900 of Figure 9 or in communication system 1000 of Figure 10. The wireless device 1100 may be alternatively referred to as a UE 1100, like a UE 912 within the context of communication system 900, or as a station (STA) 1012 or as a non-access-point station (non-AP STA) 1012, like a STA 1012 within the context of the communication system 1000, in accordance with respective embodiments. As used herein, a wireless device refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other wireless devices. Examples of a wireless device include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, and wireless terminal. Other examples include any type of UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0135] A wireless device 1100 may support device-to-device (D2D) communication, for example by implementing a 3 GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to- vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, wireless device 1100 may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, wireless device 1100 may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, wireless device 1100 may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0136] In particular embodiments, wireless device 1100 includes processing circuitry 1102 that is operatively coupled via a bus 1104 to an input / output interface 1106, a power source 1108,P112470W001a memory 1110, a communication interface 1112, and / or any other component, or any combination thereof. Certain embodiments of wireless device 1100 may include all or a subset of the components shown in Figure 11. The level of integration between the components may vary from one embodiment of wireless device 1100 to another. In general, in a particular embodiment of wireless device 1100, processing circuitry 1102, input / output interface 1106, power source 1108, memory 1110, and communication interface 1112 may, in whole or in part, represent or include physical components common to or shared by one or more of the other elements of wireless device 1100. Further, certain embodiments of wireless devices 1100 may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0137] The processing circuitry 1102 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 1110. The processing circuitry 1102 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 1102 may include multiple central processing units (CPUs).

[0138] In the example, the input / output interface 1106 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into wireless device 1100. Examples of an input device include a touch-sensitive or presence- sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0139] In some embodiments, the power source 1108 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used to supply power to circuitry or to charge anP112470W001associated battery. The power source 1108 may further include power circuitry for delivering power from the power source 1108 itself, and / or an external power source, to the various parts of wireless device 1100 via input circuitry or an interface such as an electrical power cable. Power source 1108 may perform any formatting, converting, or other modification to make accessible power suitable for the respective components of the wireless device 1100 to which power is supplied.

[0140] The memory 1110 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 1110 includes one or more programs 1114, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 1116. The memory 1110 may store, for use by wireless device 1100, any of a variety of various operating systems or combinations of operating systems.

[0141] The memory 1110 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 1110 may allow wireless device 1100 to access instructions, programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 1110, which may be or comprise a device-readable storage medium.

[0142] The processing circuitry 1102 may be configured to communicate with an access network or other network via or using the communication interface 1112. The communication interface 1112 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 1122. The communication interface 1112 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another wireless device orP112470W001a network node in an access network). Each transceiver may include a transmitter 1118 and / or a receiver 1120 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 1118 and receiver 1120 may be coupled to one or more antennas (e.g., antenna 1122) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0143] In the illustrated embodiment, communication functions of the communication interface 1112 may include cellular communication, Wi-Fi communication (e.g., according to an IEEE 802.11 family standard), LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near- field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0144] In particular embodiments, wireless device 1100 may provide an output of data captured via a sensor, through its communication interface 1112, via a wireless connection to a network node, and / or in any appropriate manner. Data captured by sensors of a wireless device 1100 can be communicated through a wireless connection to a network node via another wireless device 1100. In particular embodiments, such output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0145] As another example, wireless device 1100 comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, wireless device 1100 may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0146] Wireless device 1100, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to,P112470W001wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. In particular embodiments, wireless device 1100 represents an loT device that comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the example embodiment of wireless device 1100 shown in Figure 11.

[0147] As yet another specific example, in an loT scenario, wireless device 1100 may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another wireless device and / or a network node. Wireless device 1100 may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, wireless device 1100 may implement the 3GPP NB-IoT standard. In other scenarios, wireless device 1100 may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0148] In practice, any number of wireless devices 1100 may be used together with respect to a single use case. For example, a first wireless device 1100 might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second wireless device 1100 that is a remote controller operating the drone. When a user makes changes from the remote controller, the first wireless device 1100 may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second wireless device 1100 can also include more than one of the functionalities described above. For example, wireless device 1100 might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0149] Figure 12 shows a network node 1200 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in aP112470W001telecommunications network. In accordance with respective embodiments, network node 1200 may be configured to operate in communication system 900 of Figure 9, like network nodes 908 or 910, or in communication system 1000 of Figure 10, like an AP 1010 or a station 1012. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0150] Network nodes 1200 may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. Network node 1200 may be a relay node or a relay donor node controlling a relay. Network nodes 1200 may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0151] Other examples of network nodes 1200 include multiple transmission point (multi-TRP) 5G access nodes, multi- standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0152] In particular embodiments, network node 1200 includes a processing circuitry 1202, a memory 1204, a communication interface 1206, and a power source 1208. In general, in a particular embodiment of network node 1200, processing circuitry 1202, memory 1204, communication interface 1206, and power source 1208 may, in whole or in part, represent or include physical components common to or shared by one or more of the other elements of network node 1200.

[0153] The network node 1200 may be composed of multiple distinct network entities (e.g., a NodeB entity and a RNC entity, or a BTS entity and a BSC entity, etc.), which may each have or utilize their own respective physical components. In certain scenarios in which the network node 1200 comprises multiple such entities (e.g., BTS and BSC), one or more of the separate entities may be shared among several network nodes. For example, a single RNC may control multipleP112470W001NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 1200 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memories 1204 or portions of memory 1204 for different RATs) and some components may be reused (e.g., a same antenna 1210 may be shared by different RATs). The network node 1200 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 1200, for example GSM, WCDMA, LTE, NR, Wi-Fi (e.g., according to an IEEE 802.11 family standard), Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 1200.

[0154] The processing circuitry 1202 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application- specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other components, such as the memory 1204, to provide network node 1200 functionality.

[0155] In some embodiments, the processing circuitry 1202 includes a system on a chip (SOC). In some embodiments, the processing circuitry 1202 includes one or more of radio frequency (RF) transceiver circuitry 1212 and baseband processing circuitry 1214. In some embodiments, the RF transceiver circuitry 1212 and the baseband processing circuitry 1214 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 1212 and baseband processing circuitry 1214 may be on the same chip or set of chips, boards, or units.

[0156] The memory 1204 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 1202. The memory 1204 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 1202 andP112470W001utilized by the network node 1200. The memory 1204 may be used to store any calculations made by the processing circuitry 1202 and / or any data received via the communication interface 1206. In some embodiments, the processing circuitry 1202 and memory 1204 is integrated.

[0157] The communication interface 1206 is used in wired or wireless communication of signaling and / or data with UEs, other network nodes, and / or any other network equipment. In the illustrated embodiment, communication interface 1206 comprises port(s) / terminal(s) 1216 to send and receive data, for example to and from a network over a wired connection. In particular embodiments, network node 1200 may be capable of wireless communication and communication interface 1206 may also include radio front-end circuitry 1218 that may be coupled to, or in certain embodiments a part of, an antenna 1210. Particular embodiments of radio front-end circuitry 1218 include filter(s) 1220 and amplifier(s) 1222. The radio front-end circuitry 1218 may be connected to an antenna 1210 and processing circuitry 1202. The radio front-end circuitry may be configured to condition signals communicated between antenna 1210 and processing circuitry 1202. The radio front-end circuitry 1218 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 1218 may convert the digital data into a radio signal(s) having the appropriate channel and bandwidth parameters using a combination of filters 1220 and / or amplifiers 1222. The radio signal(s) may then be transmitted via the antenna 1210. Similarly, when receiving data, the antenna 1210 may collect radio signals which are then converted into digital data by the radio front-end circuitry 1218. The digital data may be passed to the processing circuitry 1202. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0158] In certain alternative embodiments, network node 1200 may be capable of wireless communication but does not include separate radio front-end circuitry 1218, instead, the processing circuitry 1202 includes radio front-end circuitry and is connected to the antenna 1210. Similarly, in some embodiments, all or some of the RF transceiver circuitry 1212 is part of the communication interface 1206. In still other embodiments, the communication interface 1206 includes one or more ports or terminals 1216, the radio front-end circuitry 1218, and the RF transceiver circuitry 1212, as part of a radio unit (not shown), and the communication interface 1206 communicates with the baseband processing circuitry 1214, which is part of a digital unit (not shown).

[0159] The antenna 1210 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 1210 may be coupled to the radio front-end circuitry 1218 and may be any type of antenna capable of transmitting and receiving data and / orP112470W001signals wirelessly. In certain embodiments, the antenna 1210 is separate from the network node 1200 and connectable to the network node 1200 through one or more interfaces or ports.

[0160] The antenna 1210, communication interface 1206, and / or the processing circuitry 1202 may be configured to perform some or all of the receiving operations and / or obtaining operations described herein as being performed by the network node 1200. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 1210, the communication interface 1206, and / or the processing circuitry 1202 may be configured to perform some or all of the transmitting or sending operations described herein as being performed by the network node 1200. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0161] The power source 1208 provides power to the various components of network node 1200 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 1208 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 1200 with power for performing the functionality described herein. For example, the network node 1200 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 1208. As a further example, the power source 1208 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0162] Embodiments of the network node 1200 may include additional components beyond those shown in Figure 12 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 1200 may include user interface equipment to allow input of information into the network node 1200 and to allow output of information from the network node 1200. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 1200.

[0163] Figure 13 is a block diagram illustrating a virtualization environment 1300 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or moreP112470W001virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1300 hosted by one or more of hardware nodes, such as a hardware computing device that operates as an access network node, UE, core network node, or host. Further, in embodiments in which a virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 1300 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface.

[0164] Applications 1302 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 1300 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0165] Hardware 1304 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1306 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VM 1308A and VM 1308B (which may be collectively referred to as VMs 1308), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 1306 may present a virtual operating platform that appears like networking hardware to one or more of the VMs 1308.

[0166] The VMs 1308 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by virtualization layer 1306. Different embodiments of the instance of a virtual appliance 1302 may be implemented on one or more of VMs 1308, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0167] In the context of NFV, each of the VMs 1308 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1308, and that part of hardware 1304 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, formsP112470W001separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more of the VMs 1308 on top of the hardware 1304 and corresponds to an application 1302.

[0168] Hardware 1304 may be implemented in a standalone network node with generic or specific components. Hardware 1304 may implement some functions via virtualization. Alternatively, hardware 1304 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1310, which, among others, oversees lifecycle management of applications 1302. In some embodiments, hardware 1304 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1312 which may alternatively be used for communication between hardware nodes and radio units.

[0169] Further definitions and embodiments are discussed below.

[0170] In the above-description of certain embodiments of the present disclosure, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which concepts of the present disclosure belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0171] When an element is referred to as being “connected”, “coupled”, “responsive”, or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected”, “directly coupled”, “directly responsive”, or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, “coupled”, “connected”, “responsive”, or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / orP112470W001clarity. The term “and / or” (abbreviated “ / ”) includes any and all combinations of one or more of the associated listed items.

[0172] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of concepts of the present disclosure. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0173] As used herein, the terms “comprise”, “comprising”, “comprises”, “include”, “including”, “includes”, “have”, “has”, “having”, or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components, or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof. Furthermore, as used herein, the common abbreviation “e.g.”, which derives from the Latin phrase “exempli gratia,” may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation “i.e.”, which derives from the Latin phrase “id est,” may be used to specify a particular item from a more general recitation.

[0174] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0175] These computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable mediumP112470W001produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of the present disclosure may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as “circuitry,” “a module” or variants thereof.

[0176] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of the present disclosure. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0177] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present disclosure. All such variations and modifications are intended to be included herein within the scope of present disclosure. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the present disclosure including the examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

P112470W001Claims:

1. A computer-implemented method performed by a computing device for updating a machine learning, ML, model for prediction of a metric in a network, the method comprising: accessing (700) (i) data to be incorporated into the ML model, the ML model comprising a set of data generators and a solver functionality, and (ii) a similarity metric for determination of a similarity between the data and a respective data output from respective data generators in the set of data generators;determining (702) the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators;updating (704) at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; and providing (720) an updated set of data generators comprising the at least one of the respective data generator.

2. The computer-implemented method of Claim 1 , wherein the updated set of data generators is provided for the solver functionality to predict the metric.

3. The computer-implemented method of any one of Claims 1 to 2, further comprising: creating (706) a new data generator with the data when the similarity criterion is not met for at least one of the respective data generators.

4. The computer-implemented method of any one of Claims 1 to 3, further comprising: merging (714) two respective data generators in the set of data generators, which results in a reduced number of data generators in the set of generators, when the similarity criterion for each of the two respective data generators is met.

5. The computer-implemented method of any one of Claims 1 to 4, further comprising: training (716) the solver functionality with the data from the updated set of data generators or a subset of the data generators.

6. The computer-implemented method of any one of Claims 1 to 5, further comprising: removing (718), from the set of data generators, a data generator that has learned from undesirable data.P112470W0017. The computer-implemented method of any one of Claims 1 to 6, further comprising:predicting (722) the metric with the ML model comprising the updated set of data generators and the solver functionality; andproviding (724) the predicted metric towards a device that requested the metric.

8. The computer-implemented method of any one of Claims 1 to 7, wherein the accessing (700) further comprises accessing an intent comprising a maximum number of data generators for the set of data generators.

9. The computer-implemented method of any one of Claims 3 to 8, further comprising: training (708) the ML model comprising the new data generator with the data; checking (710) whether a number of the data generators in the ML model exceeds the maximum number of data generators;when the number exceeds the maximum number of data generators, determining (712) whether the similarity criterion is met, based on the similarity metric, between the data and the respective data output from the respective data generators; andmerging (714) two respective data generators in the updated set of data generators when the similarity criterion for each of the two respective data generators is met.

10. The computer-implemented method of any one of Claims 3 to 9, wherein the merging (714) comprises merging two or more respective data generators in the set of data generators that are most similar.

11. The computer-implemented method of any one of Claims 1 to 10, wherein the similarity metric is a metric that measures a deviation between the data and the data output from a respective data generator.

12. The computer-implemented method of any one of Claims 1 to 11, wherein the data corresponds to a new task when there is at least one of a change in a network condition and a change in a data distribution.

13. The computer-implemented method of any one of Claims 1 to 12, wherein the prediction of the metric in the network comprises prediction of at least one of a one-way delay, aP112470W001round-trip time, a packet loss, a corrupted packet, a reference signal received power (RSRP), and a bit error rate (BER) in a telecommunications network.

14. The computer-implemented method of any one of Claims 1 to 13, wherein the network comprises a telecommunications network.

15. A computing device (404, 604, 608, 800, 908, 912, 1012, 1032, 1100, 1200, 1302) configured to update a machine learning, ML, model (808, 810, 814) for prediction of a metric in a network, the computing device comprising:the machine learning, ML, model comprising a set of data generators (810) and a solver functionality (814);at least one processor (802);at least one memory (804) connected to the at least one processor and storing program code (808) and / or the ML model (810, 814) that is executed by the at least one processor to perform operations comprising:access (i) data to be incorporated into the ML model and (ii) a similarity metric for determination of a similarity between the data and a respective data output from respective data generators in the set of data generators;determine the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators;update at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; and provide an updated set of data generators comprising the at least one of the respective data generators.

16. The computing device of Claim 15, wherein the at least one memory (804) is connected to the at least one processor (802) and stores program code (808) and / or the ML model (810, 814) that is executed by the at least one processor to perform operations according to any one of Claims 2 to 14.

17. A computing device (404, 604, 608, 800, 908, 912, 1012, 1032, 1100, 1200, 1302) configured to update a machine learning, ML, model (808, 810, 814) for prediction of a metric in a network, the computing device comprising:the machine learning, ML, model comprising a solver functionality (814);P112470W001at least one processor (802);at least one memory (804) connected to the at least one processor and storing program code (808) and / or the ML model (810, 814) that is executed by the at least one processor to perform operations comprising:access (i) data to be incorporated into the ML model and (ii) a similarity metric for determination of a similarity between the data and an output from respective data generators in a set of data generators;determine the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators;update at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; and provide an updated set of data generators comprising the at least one of the respective data generators.

18. The computing device of Claim 17, wherein the at least one memory (804) is connected to the at least one processor (802) and stores program code (808) and / or the ML model (814) that is executed by the at least one processor to perform operations according to any one of Claims 2 to 14.

19. A computer program product comprising a non-transitory storage medium (804) including program code (808) to be executed by at least one processor (802) of a computing device (404, 604, 608, 800, 908, 912, 1012, 1032, 1100, 1200, 1302) configured to update a machine learning, ML, model (808, 810, 814) for prediction of a metric in a network, the computer program product comprising program code (808) and / or the ML model (808, 810, 814), whereby execution of the program code and / or the ML model causes the computing device to perform operations comprising:access (i) data to be incorporated into the ML model and (ii) a similarity metric for determination of a similarity between the data and an output from respective data generators in a set of data generators;determine the similarity, based on the similarity metric, between the data and the respective data output from the respective data generators;update at least one of the respective data generators with the data when the similarity fulfills a similarity criterion for the at least one of the respective data generators; andP112470W001provide an updated set of data generators comprising the at least one of the respective data generators.

20. The computer program product of Claim 19, whereby execution of the program code and / or the ML model causes the computing device to perform operations according to any of Claims 2 to 14.

Images