Data collection for trustworthiness assessment

Abstract

Embodiments of the present disclosure relate to devices, methods, apparatuses and a computer readable storage medium related to data collection for trustworthiness assessment In an aspect, the terminal device transmits, to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics. The terminal device receives, from the network device, the data with the trustworthiness related characteristics. The terminal device performs the trustworthiness assessment based on the data.

Description

DATA COLLECTION FOR TRUSTWORTHINESS ASSESSMENTFIELD

[0001] Various embodiments generally relate to the field of communication, and in particular, to devices, methods, apparatuses and a computer readable storage medium related to data collection for trustworthiness assessment.BACKGROUND

[0002] A communication network can be seen as a facility that enables communications between two or more communication devices, or provides communication devices access to a data network. A mobile or wireless communication network is one example of a communication network.

[0003] Such communication networks operate in accordance with standards, such as those promulgated by 3GPP (Third Generation Partnership Project), IEEE or ETSI (European Telecommunications Standards Institute). Examples of such standards include the so-called 5G (5th Generation) standard or other standards promulgated by 3GPP.SUMMARY

[0004] In general, embodiments of the present disclosure provide devices, methods, apparatuses and a computer readable storage medium for communication, for example, for data collection for trustworthiness assessment, especially for data collection for assessing artificial intelligence (Al) / machine learning (ML) trustworthiness.

[0005] In a first aspect, there is provided a terminal device. The terminal device may comprise at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the terminal device at least to: transmit, to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; receive, from the network device, the data with the trustworthiness related characteristics; and perform the trustworthiness assessment based on the data.

[0006] In a second aspect, there is provided a network device. The network device may comprise at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the network device at least to: receive, from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; obtain the data with the trustworthiness related characteristics; and transmit, to the terminal device, the data with the trustworthiness related characteristics.

[0007] In a third aspect, there is provided a method. The method may comprise: transmitting, at a terminal device and to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; receiving, from the network device, the data with the trustworthiness related characteristics; and performing the trustworthiness assessment based on the data.

[0008] In a fourth aspect, there is provided a method. The method may comprise: receiving, at a network device and from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; obtaining the data with the trustworthiness related characteristics; and transmitting, to the terminal device, the data with the trustworthiness related characteristics.

[0009] In a fifth aspect, there is provided an apparatus. The apparatus may comprise: means for transmitting, at a terminal device and to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; means for receiving, from the network device, the data with the trustworthiness related characteristics; and means for performing the trustworthiness assessment based on the data.

[0010] In a sixth aspect, there is provided an apparatus. The apparatus may comprise: means for receiving, at a network device and from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; means for obtaining the data with the trustworthiness related characteristics; and means for transmitting, to the terminal device, the data with the trustworthiness related characteristics.

[0011] In a seventh aspect, there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus to perform at least the method according to the third to fourth aspects.

[0012] In an eighth aspect, there is provided a computer program comprising instructions, which, when executed by an apparatus, cause the apparatus to perform at least the method according to the third to fourth aspects.

[0013] In a tenth aspect, there is provided a terminal device. The terminal device may comprise transmitting circuitry configured to transmit, to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; receiving circuitry configured to receive, from the network device, the data with the trustworthiness related characteristics; and performing circuitry configured to perform the trustworthiness assessment based on the data.

[0014] In an eleventh aspect, there is provided a network device. The network device may comprise receiving circuitry configured to receive, from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; obtaining circuitry configured to obtain the data with the trustworthiness related characteristics; and transmitting circuitry configured to transmit, to the terminal device, the data with the trustworthiness related characteristics.

[0015] In a twelfth aspect, there is provided a terminal device. The terminal device may comprise at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the terminal device at least to: receive, from a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; and transmit, to the network device, the data with the trustworthiness related characteristics.

[0016] In a thirteenth aspect, there is provided a network device. The network device may comprise at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the network device at least to: transmit, to a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicatestrustworthiness related characteristics; receive, from the terminal device, the data with the trustworthiness related characteristics; and perform the trustworthiness assessment based on the data.

[0017] In a fourteenth aspect, there is provided a method. The method may comprise: receiving, at a terminal device and from a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; and transmitting, to the network device, the data with the trustworthiness related characteristics.

[0018] In a fifteenth aspect, there is provided a method. The method may comprise: transmitting, at a network device and to a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; receiving, from the terminal device, the data with the trustworthiness related characteristics; and performing the trustworthiness assessment based on the data.

[0019] In a sixteenth aspect, there is provided an apparatus. The apparatus may comprise: means for receiving, at a terminal device and from a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; and means for transmitting, to the network device, the data with the trustworthiness related characteristics.

[0020] In a seventeenth aspect, there is provided an apparatus. The apparatus may comprise: means for transmitting, at a network device and to a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; means for receiving, from the terminal device, the data with the trustworthiness related characteristics; and means for performing the trustworthiness assessment based on the data.

[0021] In an eighteenth aspect, there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus to perform at least the method according to the fourteenth or fifteenth aspect.

[0022] In a nineteenth aspect, there is provided a computer program comprising instructions, which, when executed by an apparatus, cause the apparatus to perform at least the method according to the fourteenth or fifteenth aspect.

[0023] In a twentieth aspect, there is provided a terminal device. The terminal device may comprise receiving circuitry configured to receive, from a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; and transmitting circuitry configured to transmit, to the network device, the data with the trustworthiness related characteristics.

[0024] In a twenty-first aspect, there is provided a network device. The network device may comprise transmitting circuitry configured to transmit, to a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; receiving circuitry configured to receive, from the terminal device, the data with the trustworthiness related characteristics; and performing circuitry configured to perform the trustworthiness assessment based on the data.

[0025] It is to be understood that the summary section is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become easily comprehensible through the following description.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Some embodiments will now be described with reference to the accompanying drawings, in which:

[0027] FIG. 1 illustrates an example of an application scenario in which some embodiments of the present disclosure may be implemented;

[0028] FIG. 2 illustrates an example communication process between a terminal device and a network device according to some embodiments of the present disclosure;

[0029] FIG. 3 illustrates an example process according to some embodiments of the present disclosure;

[0030] FIG. 4 illustrates another example process between a terminal device and a network device according to some embodiments of the present disclosure;

[0031] FIG. 5 illustrates another example process according to some embodiments of the present disclosure;

[0032] FIG. 6 illustrates a flowchart of an example method implemented at a terminal device in accordance with some embodiments of the present disclosure;

[0033] FIG. 7 illustrates a flowchart of an example method implemented at a network device in accordance with some embodiments of the present disclosure;

[0034] FIG. 8 illustrates a flowchart of an example method implemented at a terminal device in accordance with some other embodiments of the present disclosure;

[0035] FIG. 9 illustrates a flowchart of an example method implemented at a network device in accordance with some other embodiments of the present disclosure;

[0036] FIG. 10 illustrates an example simplified block diagram of a device that is suitable for implementing embodiments of the present disclosure; and

[0037] FIG. 11 illustrates an example block diagram of an example computer readable medium in accordance with some embodiments of the present disclosure.

[0038] Throughout the drawings, the same or similar reference numerals represent the same or similar element.DETAILED DESCRIPTION

[0039] Principles of the present disclosure will now be described with reference to some embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. The disclosure described herein may be implemented in various manners other than the ones described below.

[0040] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which the present disclosure belongs.

[0041] References in the present disclosure to “one embodiment,” “an embodiment,” “an embodiment,” and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is withinthe knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0042] It may be understood that although the terms “first”, “second”, “third”, “fourth” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0043] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof. As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements. Additionally, the terms “identity”, “identifier” and “identification” in the present disclosure have the same meaning and could be collectively represent by “ID”.

[0044] As used in this application, the term “circuitry” may refer to one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analog and / or digital hardware circuit(s) with software / firmware and(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and(c) hardware circuit(s) and or processor(s), such as a microprocessor s) or a portion of a microprocessor s) that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0045] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0046] As used herein, the term “communication network” refers to a network following any suitable communication standards, such as new radio (NR), long term evolution (LTE), LTE-advanced (LTE-A), wideband code division multiple access (WCDMA), high-speed packet access (HSPA), narrow band Internet of things (NB-IoT) and so on. Furthermore, the communications between a terminal device and a network device in the communication network may be performed according to any suitable generation communication protocols, including, but not limited to, the third generation (3G), the fourth generation (4G), 4.5G, the fifth generation (5G) communication protocols, the sixth generation (6G) communication protocols, and / or beyond. Embodiments of the present disclosure may be applied in various communication systems. Given the rapid development in communications, there will of course also be future type communication technologies and systems with which the present disclosure may be embodied. It should not be seen as limiting the scope of the present disclosure to only the aforementioned system.

[0047] As used herein, the term “network device” refers to a node in a communication network via which a terminal device accesses the network and receives services therefrom. The network device may refer to a base station (BS) or an access point (AP), for example, a node B (NodeB or NB), an evolved NodeB (eNodeB or eNB), a NR NB (also referred to as a gNB), a transmit-receive point (TRP), a remote radio unit (RRU), a radio header (RH), a remote radio head (RRH), a relay, a low power node such as a femto, a pico, an Integrated Access and Backhaul (IAB) node, a non-terrestrial network (NTN) or non-ground network device such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous earth orbit (GEO) satellite, an aircraft network device, and so forth,depending on the applied terminology and technology. In some embodiments, radio access network (RAN) split architecture comprises a Centralized Unit (CU) and a Distributed Unit (DU) at an IAB donor node. An IAB node comprises a Mobile Terminal (IAB-MT) part that behaves like a UE toward the parent node, and a DU part of an IAB node behaves like a base station toward the next-hop IAB node.

[0048] The term “terminal device” refers to any end device that may be capable of wireless communication. By way of example rather than limitation, a terminal device may also be referred to as a communication device, user equipment (UE), a subscriber station (SS), a portable subscriber station, a mobile station (MS), or an access terminal (AT). The terminal device may include, but not limited to, a mobile phone, a cellular phone, a smart phone, voice over IP (VoIP) phones, wireless local loop phones, a tablet, a wearable terminal device, a personal digital assistant (PDA), portable computers, desktop computer, image capture terminal devices such as digital cameras, gaming terminal devices, music storage and playback appliances, vehicle-mounted wireless terminal devices, wireless endpoints, mobile stations, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), USB dongles, smart devices, wireless customer-premises equipment (CPE), an Internet of things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial, a relay node, an integrated access and backhaul (IAB) node, and / or industrial wireless networks, and the like. In the following description, the terms “terminal device”, “communication device”, “terminal”, “user equipment” and “UE” may be used interchangeably.

[0049] As used herein, the term “resource” may refer to any resource for performing a communication, for example, a communication between a terminal device and a network device, such as a resource in time domain, a resource in frequency domain, a resource in space domain, a resource in code domain, a resource in a combination of more than one domain or any other resource enabling a communication, and the like.

[0050] The application of artificial intelligence (Al) / machine learning (ML) to wireless communications has been thus far limited to implementation-based approaches, both, at the network and the UE sides. However, augmenting the air-interface with features enabling improved support of AI / ML based algorithms may potentially offer enhanced performancee.g., improved throughput, robustness, accuracy or reliability, etc. depending on the use cases as well as reduced complexity / overhead. To this end, 3GPP is now actively pursuing AI / ML for Air Interface as captured in the following.

[0051] In an aspect of 3GPP discussions on AI / ML for air interface, 3GPP has first time started studying AI / ML use cases in RAN in Release 18, which is ongoing with the work item (WI) in Release 19, called artificial intelligence (Al) / machine learning (ML) for new radio (NR) air interface. It has considered three use cases: (i) CSI feedback enhancement, e.g., overhead reduction, improved accuracy, prediction; (ii) Beam management, e.g., beam prediction in time, and / or spatial domain for overhead and latency reduction, beam selection accuracy improvement; (iii) Positioning accuracy enhancements for different scenarios including, e.g., those with heavy NLOS conditions. In addition to studying performance benefits of above use cases in various scenarios, for specification purposes, study item (SI) / WI has focused on essential aspects of AI / ML framework, namely, inference, monitoring, data collection, and training, which are collectively referred to as life cycle management (LCM). Some details of the general AI / ML framework are identified as TR 38.843.

[0052] Some embodiments of the present disclosure may be related to AI / ML use cases to be supported by RAN in 6G. In order to describe the embodiment of the present disclosure more clearly, some contents related to AI / ML trustworthiness will be described as below. Explainability, fairness, and robustness are the key factors that contribute to trustworthy AI / ML.

[0053] For the first key factor, i.e. the explainability, in the context of explainable AI / ML, explanations can be categorized into two types: (1) Local explanation that aims to explain individual predictions by the ML model, i.e., it focuses on explaining why a specific prediction was made by the ML model for a particular input; (2) Global explanation that aims to explain the whole ML model behaviour, i.e., it focuses on explaining how the ML model works in general across several / all possible inputs. Both are important for ensuring trustworthiness in AI / ML solutions as well as maintaining regulatory compliance.

[0054] There are three broad approaches to design an ML model to be explainable: (a) Premodelling explainability - Aims to understand training data that was used to train an AI / ML model. For example, using state of the art algorithms such as ProtoDash and Disentangled Inferred Prior VAE; (b) Explainable modelling / Interpretable modelling - Aims to train an explainable AI / ML model, e.g., train an ML model that can generate joint prediction andexplanation or train another surrogate explainable ML model together with the complex neural network ML model. For example, using state of the art algorithms such as Generalized Linear Rule Models and Teaching Explainable Decisions (TED); (c) Post-modelling explainability - Aims to extract explanations from pre-trained AI / ML models during inference phase. For example, using state of the art algorithms such as ProtoDash, Contrastive Explanations Method, Profweight, LIME and SHAP.

[0055] In an aspect of quantification of explainability, although it is ultimately the consumer who determines the quality of an explanation, the research community has proposed quantitative metrics as proxies for explainability. The following are some of the examples.

[0056] Feature importance spread, feature importance stability, predictions groups contrast, alpha-feature importance, explainability ease score, and surrogacy efficacy score.

[0057] For the second key factor, i.e. the fairness, specialized terminology from the field of fairness in AI / ML has been defined in some solutions. A favorable label is a label whose value corresponds to an outcome that provides an advantage to the recipient. A protected attribute is an attribute that partitions a population into groups that have parity in terms of benefit received. Protected attributes are not universal but are application specific. A privileged value of a protected attribute indicates a group that has historically been at a systematic advantage. Group fairness is the goal of groups defined by protected attributes receiving similar treatments or outcomes. Individual fairness is the goal of similar individuals receiving similar treatments or outcomes. Bias is a systematic error. In the context of fairness, we are concerned with unwanted bias that places privileged groups at a systematic advantage and unprivileged groups at a systematic disadvantage. A fairness metric is a quantification of unwanted bias in training data or models. A bias mitigation algorithm is a procedure for reducing unwanted bias in training data or models. Therefore, it is important to apply fairness analysis (i.e., for detecting, understanding and mitigating unwanted algorithmic bias) throughout the AI / ML Pipeline, making sure to continuously reevaluate the models from the perspective of fairness. This is especially important when AI / ML is deployed in critical telco processes (e.g., AI / ML based positioning, mobility management) that affect a wide range of end users (e.g., UEs, verticals).

[0058] There are three broad approaches to achieve the goal of fair predictions in the AI / ML model: (1) Pre-processing fairness (to detect bias in the AI / ML training data); (2) In-processing fairness (to detect bias during the AI / ML model training); (3) Post-processing fairness (to detect bias in the AI / ML model decisions).

[0059] There are several metrics that measure individual and group fairness. For example: 1) Statistical Parity Difference, i.e. the difference in the rate of favourable outcomes received by the unprivileged group to the privileged group; 2) Average Odds Difference, i.e. the average difference of false positive rate and true positive rate between unprivileged and privileged groups; 3) Disparate Impact, i.e. the ratio of the rate of a favourable outcome for the unprivileged group to that of the privileged group; 4) Theil Index which measures the inequality in benefit allocation for individuals.

[0060] For the third key factor, i.e. the robustness, e.g. the robustness to adversarial attacks. Various types of adversarial attacks on AI / ML models have been described in some solutions. Some most important concepts / terminologies on adversarial attacks will be introduces below.

[0061] Adversarial attacks are studied using a variety of threat models. The two most common threat models are the whitebox and blackbox threat models. In the whitebox threat model, an adversary has visibility into the model parameters including, but not limited to the architecture, weights, pre- and post- processing steps. The whitebox threat model is thought to represent the strongest attacker as they have full knowledge of the system. In the blackbox threat model, the adversary only has query access to the model. That is to say, given an input from the adversary, the model provides either a soft output (i.e., prediction probabilities) or a hard output (i.e., top-1 or top-k output labels). Blackbox attacks are perceived as the realistic threat model when evaluating a system for deployment. There are four broad categories of adversarial attacks on AI / ML models, as follows.

[0062] Adversarial Evasion Attack, which is an inference time attack in which the adversary seeks to add adversarial noise to an input and create an adversarial sample. These samples, when provided to a well-trained target model, cause predictable errors at the model's output. Evasion attacks can be targeted (i.e., the noise causes a specific error at the output) or untargeted (i.e., the noise causes an error at the output, but the type of error is not important to the adversary). Evasion attacks can be classified into four types: (i) Gradient-based attack is a type of whitebox attack where the attacker uses the model's gradient with respect to the adversarial object in order to identify the optimal adversarial noise to add; (ii) Confidence score attack is a type of blackbox attack where the attacker uses the outputted classification confidence to estimate the gradients of the model, and then perform similar smartoptimization to gradient-based attacks above; (iii) Hard label attack is a type of blackbox attack that rely solely on the label outputted by the model and don’t require the confidence scores. This makes the attack dumber but also more realistic; (iv) Surrogate model attack is very similar to gradient-based attacks, except they require an extra step. When the attacker doesn’t have access to the model’s internals but still wants to mount a whitebox attack, they can try to first rebuild the target’s model, e.g., by repeatedly querying the target model and observing input-output pairs. Then, identify the optimal adversarial noise to add by performing whitebox attack on the substitute model and transfer the learning to attack the target blackbox model.

[0063] Adversarial Poisoning Attack, which is a training time attack in which the adversary uses direct or indirect methods to corrupt the training data in order to achieve a specific goal. Poisoning is a major concern whenever the adversary has the ability to influence the training data, such as in online learning, in which live data is periodically used to retrain the model so as to remain robust to concept drift. Through poisoning, an adversary can degrade model performance and inject backdoors into the model so as to induce certain errors when triggered. Poisoning is considered as an integrity attack because tampering with the training data impacts the model's ability to output correct predictions.

[0064] Adversarial Inference & Inversion Attacks, which are are inference time attacks in which the adversary uses API access to a target blackbox model in order to extract information about the training data. In a model inference attack (e.g., confidence-based attack, label-based attack), the adversary uses the API in order to learn the data distribution of the training data or determine if certain data points were used when training the target model. In an adversarial inversion attack, the adversary uses the API in an attempts to reconstruct a training data sample from its confidence score vector predicted by the target model. Adversarial inference is a major issue when the confidentiality of the data needs to be maintained due to privacy or proprietary reasons.

[0065] Adversarial Model Extraction (or Model Stealing) Attack, which is an inference time attack in which the adversary uses API access to the target blackbox model in order to learn the target model's parameters or create an approximation of the target model. By querying the model and using the outputs as the labels along with their confidence scores, the adversary can train a new, substitute model whose performance is similar to the target model. Once trained, the adversary can re-use the model for their own purposes (theft) or perform evasion attacks on the substitute model, which can then be transferred to the target model with highlikelihood to succeed. Adversarial model extraction is a major issue when the confidentiality of the model needs to be maintained due to intellectual property rights.

[0066] In an aspect of adversarial defences for AI / ML models, for each category of adversarial attacks introduced above, there exist various defence mechanisms. They are adversarial evasion defence, adversarial poisoning defence, adversarial inference & inversion defences, and adversarial model extraction defence as follows.

[0067] For the adversarial evasion defence, there are four popular types of evasion defences: (i) Adversarial training is when the defender retrains the model with adversarial examples included in the training pool, but labelled with correct labels. This teaches the model to ignore the noise and only learn from “robust” features; (ii) Gradient masking is a way to hide the gradients of the model on the assumption that gradients are needed to compute powerful attacks on models. For example, defensive distillation is one popular method to mask gradients of the model; (iii) Input modification is when an input, before being passed to the model, is in some way ‘cleaned’ to get rid of adversarial noise; (iv) Extra (NULL) class is an approach where instead of forcing the model to guess the label when it clearly doesn’t know what it is - give it the option of abstaining. This is based on the assumption that models are trained on a very particular data distribution and by definition are clueless when taken outside the bounds of that.

[0068] For the adversarial poisoning defence, a number of methods have been proposed to defend models against poisoning attacks such as detection of poisoned training data based on activations analysis, based on data provenance, based on spectral signatures.

[0069] For the adversarial inference & inversion defences, a number of studies make use of various regularization techniques and ensemble learning to reduce overfitting as a defense against inference attack. When a model overfits on training data (i.e., members), it behaves more confidently on their training data than others. As a result, the confidence scores of the model on members and non-members present different patterns, which enables the attacker to distinguish them. There are various defence methods to reduce overfitting: L2-Regularizer, Dropout, Min-Max game and Model Stacking. While most existing defenses focus on reducing overfitting to mitigate inference attack, there are also approaches proposed from different angles such as MemGuard (turns the confidence score vector into an adversarial example to fool the attacker’s membership classifier) and Differential Privacy (adds noiseeither to the obj ective function of the model or to the gradient of the model during minimizing the objective function).

[0070] For the adversarial model extraction defence, the goal of the defender is to prevent the attacker from stealing private information or replicating the model’s functionality. Specifically, given a certain budget of the attacker, the defender aims to reduce the accuracy of the stolen model established by the attacker. The defender tries to increase the cost of stealing the model to reach a certain accuracy target, thus the attacker is discouraged from performing the attack. There are two defence mechanisms: defending by output perturbation (i.e., injecting special perturbations to model predictions to obfuscate provided information such as prediction labels and confidence scores) and detecting by observing the queries (i.e., monitoring the queries from the clients and generating a warning if malicious behaviors are detected).

[0071] There are several measurable adversarial robustness metrics such as loss sensitivity, empirical robustness, clever and pointwise differential training privacy.

[0072] The current AI / ML framework considered for RAN use cases in 3 GPP (Rel. 18 SI and Rel. 19 WI on AI / ML for Air Interface) focuses on fundamental aspects of AI / ML, such as training, inference and performance monitoring, together with a functionality-based life cycle management (LCM). The framework, however, lacks aspects related to trustworthiness of AI / ML such as with regards to explainability, fairness, and adversarial robustness. Current procedures agreed in Rel. 19 WI to support data collection lack the requirements related to ML trustworthiness. E.g., in positioning use case, UE may receive data for monitoring the performance of an AI / ML functionality (that uses a UE-side model), which consists of measurements (e.g., DL PRS measurements) together with corresponding ground truth (GT) labels (e.g., UE location coordinates). Measurements and GT labels can be further accompanied by quality metrics (definitions still under discussion), which however do not consider any characteristics that are relevant to assess the trustworthiness of the AI / ML models or functionalities. In particular, as outlined above, various approaches can be utilized to assess explainability, fairness, and adversarial robustness of AI / ML models, which, in turn, necessitate different characteristics of the corresponding data to be used for their assessment.

[0073] In view of these analysis and considerations, a new solution is proposed in some embodiments of the present disclosure, in which a mechanism for collecting data with characteristics enabling efficient assessment of trustworthiness of UE-side (and / or network-side) AI / ML models or functionalities is proposed. For illustrative purposes, principles and embodiments of the present disclosure will be described below with reference to the FIGS below. However, it is to be noted that these embodiments are given to enable the skilled in the art to understand inventive concepts of the present disclosure and implement the solution as proposed herein, and not intended to limit scope of the present application in any way.

[0074] FIG. 1 illustrates an example of an application scenario in which some embodiments of the present disclosure may be implemented. The architecture 100 includes at least one terminal device and at least one network device. A terminal device 102 and a network device 104 are shown as examples. In some examples, an AI / ML model may be deployed at the terminal device 102 side. Alternatively or additionally, an AI / ML model may be deployed at the network device 104 side. In some examples, the terminal device 102 and the network device 104 may communicate with each other for a purpose of AI / ML trustworthiness assessment. In some examples, the network device 104 may communicate with other entities, e.g. a further terminal device for the purpose of AI / ML trustworthiness assessment.

[0075] The current AI / ML framework considered for RAN use cases in 3 GPP focuses on fundamental aspects of AI / ML, such as training, inference and performance monitoring, together with a functionality -based life cycle management (LCM). The framework, however, lacks aspects related to trustworthiness of AI / ML, e.g., with respect to its local explainability, fairness and robustness.

[0076] The provisioning of local explanations for UE-side ML models could have 3-fold benefits which is currently lacking: (i) it aids the UE vendor, who is the owner of the ML model, to debug the ML model using local explanations to understand the model’s behavior and identify any potential issues; (ii) it aids the network vendor in overseeing the performance of the UE-side ML model by understanding how the ML model makes its predictions and thus ensuring that the ML model is functioning as expected and intervene if required; and (iii) it facilitates regulatory compliance by preserving the local explanations together with the ML corresponding action executed. This record-keeping can be crucial for audits and for ensuring transparency and accountability in the use of ML models. In simple terms, an AI / ML model needs to be able to explain the predictions it makes, to the relevant entities, e.g., network. Such explainability can uncover any biases in the models, diagnose the errors made by them, and assist in refining, debugging, and developing improved models, and also predict and prevent any failures. Furthermore, in safety-critical applications such as positioning, it can be used to ensure safety and reliability of the system as well as to improve it to become morerobust and resilient.

[0077] Ensuring ML fairness in UE-side ML models is crucial for both UE vendors and network vendors. For UE vendors, identifying and mitigating biases in ML models ensures that different scenarios: e.g., 1) deployment scenarios such as urban or rural, large or small factory, different geographical area, etc. which may also reflect different socio-economic conditions of the society; 2) different radio characteristics incl. radio conditions (e.g., line- of-sight (LOS) / non-line-of-sight (NLOS) condition); interference scenarios, network conditions (e.g., network configuration such as PRS configuration); 3) vendor specific device or network equipment differences, etc. are treated fairly, fostering trust. For network vendors, monitoring these ML models for fairness helps maintain the integrity and reliability of the network, ensuring that no aforementioned scenario is unfairly (dis)advantaged.

[0078] ML adversarial robustness enable UE-side ML models to be more secure and reliable, ensuring that they perform well in the face of adversarial attacks, e.g., harmful devices spoofing or jamming network equipment or signals.

[0079] It becomes thus critical to incorporate the trustworthiness aspect of AI / ML into the next generation of RAN considering potential AI / ML use cases to be supported (e.g., mobility, radio resource management, etc.), as well as to fulfill the promise of an “Al-native network” in 6G. In the proposed solution of some embodiments of the present disclosure, the NW may be enabled to assess AI / ML trustworthiness of the functionalities (or models) and take LCM decision based on the assessment. In this regard, the UE is provided with assistance from the network on trustworthiness evaluation and is configured report the evaluation outcome to the network. In some embodiments of the present disclosure, the proposed solution can also be applied for the use case of NW-side ML models, and can also bring the advantages as mentioned above. Details of some examples of the present disclosure will be further described in the following embodiments, as shown in FIGS. 2 to 9.

[0080] In some embodiments herein, the term “functionality / model” means “functionality and / or model”. “Functionality” and “model” may be used interchangeably in some examples. “Functionality-based” and “model-based” may be used interchangeably in some examples.

[0081] FIG. 2 illustrates an example communication process 200 between a terminal device 202 and a network device 204 according to some embodiments of the present disclosure. Examples of the terminal device 202 may be a UE. Examples of the network device 204 may be an access network device, e.g. a gNodeB (gNB), or a core network (CN) device, e.g. alocation management function (LMF). In some examples, network device 204 may be other entities or CN devices, e.g. an access and mobility management function (AMF), Administration and Maintenance (0AM), etc.

[0082] In the process 200, the terminal device 202 transmits (210), to the network device 204, an indication 205 (or referred to as a first indication 205) for requesting data 215 for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics. Then the network device 204 may receive (220), from the terminal device 202, the indication 205 for requesting the data 215 for the trustworthiness assessment. In some examples, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML). In other words, the model may be an AI / ML model. A functionality may be an AI / ML functionality.

[0083] After receiving the indication 205, the network device 204 may obtain (230) the data 215 with the trustworthiness related characteristics. In some examples, the network device 204 may obtain the data 215 in various ways. For example, the network device 204 may generate the data 215 based with the trustworthiness related characteristics. Alternatively or additionally, the network device 204 may transmit, to a second terminal device (e.g. UE2 306 in FIG. 3 below), an indication (may be referred to as a fourth indication) for requesting the data 215, in which the fourth indication indicates the trustworthiness related characteristics, and then the network device 204 receives, from the second terminal device, the data 215 with the trustworthiness related characteristics. The procedure of the network device 204 obtaining the data 215 from the second terminal device are similar to the procedure of network device 404 obtaining data 415 from terminal device 402 as shown in FIG. 4, as such, the procedure may also refer to the operations 410, 420, 430, and 440 in the process 400 hereinafter.

[0084] Then the network device 204 may transmit (240), to the terminal device 202, the data 215 with the trustworthiness related characteristics. On the terminal device 202 side, the terminal device 202 may receive (250), from the network device 204, the data 215 with the trustworthiness related characteristics. Then the terminal device 202 may perform (260) the trustworthiness assessment based on the data 215. In some examples, the trustworthiness assessment is performed based on receiving, from the network device 204, an explicit request for performing the trustworthiness assessment. Alternatively or additionally, the trustworthiness assessment is performed based on a trigger (referred to as a first trigger) configured by the network device 204 for performing the trustworthiness assessment.Alternatively or additionally, the trustworthiness assessment is performed based on a trigger (referred to as a second trigger) predetermined by the terminal device 202 for performing the trustworthiness assessment. In some examples, the first trigger is associated with a time interval for performing the trustworthiness assessment periodically, or criteria for triggering the trustworthiness assessment to be performed, or both of them.

[0085] In some examples, the trustworthiness assessment comprises explainability assessment, fairness assessment, or adversarial robustness assessment, or any combination thereof. In some examples, the data 215 may comprise at least one measurement of at least one reference signal (such as Positioning Reference Signal (PRS), Sounding Reference Signal (SRS), Channel State Information Reference Signal (CSI-RS), Demodulation Reference Signal (DMRS), Phase Tracking Reference Signal (PTRS), primary synchronization signal (PSS) / Secondary synchronization signal (SSS), etc.). Alternatively or additionally, the data 215 may comprise at least one ground truth label or reward associated with the at least one measurement. In some examples, the trustworthiness related characteristics may comprise whether the data is to be used for the trustworthiness assessment. Alternatively or additionally, the trustworthiness related characteristics may comprise a number or a ratio of data samples for the trustworthiness assessment. Alternatively or additionally, the trustworthiness related characteristics may comprise a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0086] In some examples, the network device 204 may transmit, to the terminal device 202, configuration information for reporting at least one outcome of the trustworthiness assessment. Accordingly, the terminal device 202 may receive, from the network device 204, the configuration information for reporting at least one outcome of the trustworthiness assessment. In some examples, the trustworthiness assessment is performed further based on configuration information from the network device 204. In some examples, the configuration information may comprise information on how terminal device 202 is to assess the trustworthiness and report the at least one outcome. For example, the configuration information may comprise metrics that need to be calculated by the terminal device 202 for the trustworthiness assessment, e.g. metrics associated with explainability, fairness, or adversarial robustness, such as ratio of correct explanations over all generated explanations, or accuracy of explanations based on a similarity metric between inference output and GT, disparate impact (based on protected attribute(s) and privileged groups), and / or ratio of correct estimations with adversarial samples, etc.

[0087] In some examples, the terminal device 202 may report, to the network device 204, at least one outcome of the trustworthiness assessment. The network device 204 may receive, from the terminal device 202, the at least one outcome of the trustworthiness assessment. In some examples, the at least one outcome of the trustworthiness assessment may comprise at least one outcome of explainability assessment, at least one outcome of fairness assessment, at least one outcome of adversarial robustness assessment, or any combination thereof.

[0088] In some examples, the network device 204 may determine, based on at least one outcome of the trustworthiness assessment, a life cycle management (LCM) decision, in which the LCM decision is associated with the at least one model or the at least one functionality. Then the network device 204 may transmit, to the terminal device 202, an indication (referred to as a second indication) of the LCM decision based on the at least one outcome of the trustworthiness assessment. On the terminal device 202 side, the terminal device 202 may receive, from the network device 204, the second indication of the LCM decision based on at least one outcome of the trustworthiness assessment.

[0089] In some other examples, the terminal device 202 may determine, based on at least one outcome of the trustworthiness assessment, an LCM decision, in which the LCM decision is associated with the at least one model or the at least one functionality. Then the terminal device 202 may transmit, to the network device 204, an indication (referred to as a third indication) of the LCM decision based on the at least one outcome of the trustworthiness assessment. On the network device 204, the network device 204 receives, from the terminal device 202, the third indication of the LCM decision.

[0090] The examples as described in the process 200 may be applied for a use case in which the at least one model or at least one functionality is deployed at terminal device side, e.g. UE side. Currently, there is no mechanism in place for a UE to collect data with characteristics enabling efficient assessment of trustworthiness of UE-side AI / ML models or functionalities. The process 200 above may be applied as a procedure of collecting data with characteristics enabling efficient assessment of trustworthiness of UE-side AI / ML models or functionalities. An example method described below may enable data collection to realize efficient assessment of UE-side AI / ML models or functionalities with regards to their trustworthiness, namely in terms of explainability, fairness, and adversarial robustness, considering RAN use cases.

[0091] The example method may comprise key steps as below, in which the UE is anexample of the terminal device 202 and the NW e.g. a gNB or an LMF is an example of the network device 204. In step 1, the UE requests data (such as measurements with corresponding ground truth labels or rewards) from the network (NW) for assessing trustworthiness of UE-side AI / ML models or functionalities, with an indication on requirements on the data characteristics, which may contain at least one or more of: (a) Indication whether the data will be used for assessing explainability, fairness, or adversarial robustness of a certain AI / ML model or an AI / ML-enabled functionality or feature; (b) Number / ratio of rule-based and / or example-based (including counterfactual or contrastive) samples for explainability assessment; (c) Number / ratio of samples with certain features for fairness assessment, e.g., 90% of samples consisting of LOS measurements; (d) Number / ratio of adversarial data samples for robustness assessment, e.g., samples that consist of slightly different model input (e.g., DL PRS measurement samples with few values missing or slightly different values) that will not change the model output (e.g., UE location), or samples that consist of slightly different model input that will change the model output. Difference between data samples can be indicated via pre-defined similarity metrics, e.g., via Euclidian distance. In an embodiment, the network may give the characteristics of real adversarial data samples to the UE, and the UE can then generate the synthetic adversarial data based on the indicated characteristics.

[0092] In step 2, the NW, based on the request, generates or collects necessary data with required characteristics, and provides it to the UE. For this, the NW may configure other UEs and make use of any other procedures / methods to collect measurements and associated ground truth information or rewards. For example, for a positioning use case, the NW may configure a positioning reference unit (PRU), which is a UE with known location, to collect DL PRS measurements and associated UE location information as a ground truth (GT) label with desired characteristics. The NW may request the UE to report the outcome of its ML trustworthiness assessment, such as based on pre-defined metrics. For example, some metrics associated with explainability, fairness, or adversarial robustness, such as ratio of correct explanations over all generated explanations, or accuracy of explanations based on a similarity metric between inference output and GT, disparate impact (based on protected attribute(s) and privileged groups), and / or ratio of correct estimations with adversarial samples, etc..

[0093] In step 3, the UE, using the collected data, performs ML trustworthiness assessment and informs NW about its outcome, which may consist of: (i) Outcome of explainabilityassessment, e.g., ratio of correct explanations over all generated explanations, or accuracy of explanations based on a similarity metric between inference output and GT; (ii) Outcome of fairness assessment, e.g., disparate impact (based on protected attribute(s) and privileged groups); (iii) Outcome of adversarial robustness assessment, e.g., ratio of correct estimations with adversarial samples.

[0094] In step 4, the UE or the NW may take further actions depending on the outcome, such as taking LCM decisions, e.g., activate / deactivate / switch / fallback AI / ML model and / or functionality.

[0095] Details of steps 1-4 of the example method above may further refer to FIG. 3 below. FIG. 3 illustrates an example process 300 according to some embodiments of the present disclosure. In the process 300, a UE (represented as UE1) 302, a UE (represented as UE2) 306 and a network device (referred to as NW) 304 are involved. The UE1 302 may be an example of the terminal device 202. The NW 304 may be an example of the network device 204. As an example, the UE2 306 may be a positioning reference unit (PRU).

[0096] In operations of the process 300, in some examples, prior to the operation 301, there may be the following operation (not shown in FIG. 3) performed: the UE (e.g. UE1 302) is triggered to perform ML trustworthiness (ML TW) assessment for an AI / ML model or an AI / ML functionality. The ML TW assessment may be an example of the trustworthiness (TW) assessment associated with at least one model or at least one functionality. The AI / ML model may be an example of the at least one model. The AI / ML functionality may be an example of the at least one functionality.

[0097] The trigger for performing the ML TW assessment may be based on an explicit request for performing the trustworthiness assessment, e.g. an explicit request from the network 304 to perform TW assessment. Alternatively, the trigger for performing the ML TW assessment may be based on a first trigger configured by the network device for performing the trustworthiness assessment. As an example, the UE (i.e. the UE1 302) may be configured by the network 304 to periodically perform the TW assessment (with certain time interval). As another example, the network 304 may provide the UE (i.e. the UE1 302) with the TW assessment triggering criteria (i.e. the criteria for triggering the trustworthiness assessment to be performed), e.g., up on entering certain geographical area where there is high risk of adversarial attacks. Alternatively, the trigger for performing the ML TW assessment may be based on a second trigger predetermined by the terminal device (i.e. the UE1 302 in thisexample) for performing the trustworthiness assessment. For example, the UE (i.e. the UE1 302) may, by UE implementation, autonomously determine to trigger (i.e. via the second trigger) the TW assessment.

[0098] At 301, the UE (i.e. the UE1 302) requests data (such as measurements with corresponding ground truth labels) from the network (NW) 304 for assessing trustworthiness of UE-side AI / ML models or functionalities, by indicating the requirements on the data samples (i.e. the data). In some examples, the requirements on the data samples may be trustworthiness related characteristics, and may comprise whether the data is to be used for the trustworthiness assessment. For example, the requirements on the data samples or the trustworthiness related characteristics may comprise an indication indicating whether the data will be used for the assessing trustworthiness, which could be explainability, fairness, or adversarial robustness of a certain AI / ML model or an AI / ML-enabled functionality or feature, or any combination thereof. In some examples, the requested data may comprise rewards instead of the ground truth labels.

[0099] Alternatively and additionally, the trustworthiness related characteristics may comprise a number or a ratio of data samples for the trustworthiness assessment. For example, the number or a ratio of data samples for the trustworthiness assessment may comprise: (1) a number / ratio of rule-based and / or example-based (including counterfactual or contrastive) samples for explainability assessment, (2) a number / ratio of samples with certain features for fairness assessment, e.g., 90% of samples consisting of LOS measurements, or (3) a number / ratio of adversarial data samples for robustness assessment, e.g., samples that consist of slightly different model input (e.g., DL PRS measurement samples with few values missing or slightly different values) that will not change the model output (e.g., UE location), or samples that consist of slightly different model input that will change the model output. Difference between data samples may be indicated via pre-defined similarity metrics, e.g., via a Euclidian distance. In some examples, the number or the ratio of data samples for the trustworthiness assessment may comprise any combination of (1), (2) and (3) above.

[0100] Alternatively and additionally, the trustworthiness related characteristics may comprise a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment. For example, the trustworthiness related characteristics may comprise a number / ratio of synthetic data samples for each of above parts of the dataset or the whole dataset. The dataset, i.e. the request data. In some examples, the UE (i.e. the UE1 302) indicates the maximum tolerable number / ratio of synthetic data that can be used towardsassessing trustworthiness of UE-side model. In some examples, the UE (i.e. the UE1 302) may, by implementation, generate synthetic adversarial data samples by itself. However, in some examples, due to the lack of awareness on the actual / real attacks, the generation of such data may not reflect the realistic scenarios. Hence, the network 304 could assist the UE 302 by collecting (from other UEs (e.g. the UE2 306 etc.) who have been attacked) and providing the real adversarial data samples to the UE 302. In some examples, where the UE (i.e. the UE1 302) generates the adversarial data samples by itself, the network 304 may give the characteristics of real adversarial data samples to assist the UE 302 in generating the synthetic adversarial data. For example, the network 304 may provide the characteristics of the noise (probability density function (PDF) or mean value of the PDF) that needs to be intentionally added to the measurement to generate the adversarial data.

[0101] Then the NW 304, based on the request, may generate or collect necessary data with required characteristics (i.e. trustworthiness related characteristics). Specifically, in some examples, at 303a, the NW 304 may generate the data. Alternatively, in some other examples, at 303b and 305b, the NW 304 may collect the data from other terminal devices, e.g. the UE2 306. For example, the NW 304 may configure other UEs (e.g. the UE2 306) and make use of any other procedures / methods to collect measurements and associated ground truth information, as shown in sub-steps 303b and 305b. At 303b, the NW 304 may transmit, to the UE2 306 (an example of the second terminal device in the process 200), an indication (i.e. the fourth indication as mentioned in the process 200) for requesting the data, and the fourth indication indicates the trustworthiness related characteristics. The data may comprises measurements and / or associated ground truth labels, as shown at 303b. At 305b, the UE2 306 may provide the requested data with indicated characteristics (i.e. trustworthiness related characteristics). In some examples, for a positioning use case, the NW 304 may configure a positioning reference unit (PRU) which is a UE with known location to collect DL PRS measurements and associated UE location information as the GT label with the desired characteristics. The PRU is an example of the UE2 306.

[0102] At 307, the NW 304 provides the requested data to UE 302, and configures the UE 302 to report the outcome of its ML trustworthiness assessment, such as based on pre-defined metrics. Examples of the metrics are described at 311, such as the following metrics associated with explainability, fairness, or adversarial robustness: ratio of correct explanations over all generated explanations, or accuracy of explanations based on a similarity metric between inference output and GT, disparate impact (based on protectedattribute(s) and privileged groups), and / or ratio of correct estimations with adversarial samples, etc. In some examples, the NW 304 may indicate the above characteristics of the provided data as per request in the operation 301 (e.g., the NW 304 indicates the number / ratio of the synthetic data and / or the method used to generate synthetic data).

[0103] At 309, the UE 302, using the collected data (i.e. the received data from the NW 304), performs the ML trustworthiness assessment and derives its outcome.

[0104] At 311, the UE 302 reports its outcome of the ML trustworthiness assessment to the NW 304. In some examples, the outcome of the ML trustworthiness assessment may comprise outcome of explainability assessment, e.g., ratio of correct explanations over all generated explanations, or accuracy of explanations based on a similarity metric between inference output and GT. In some examples, the outcome of the ML trustworthiness assessment may comprise outcome of fairness assessment, e.g., disparate impact (based on protected attribute(s) and privileged groups). In some examples, the outcome of the ML trustworthiness assessment may comprise outcome of adversarial robustness assessment, e.g., ratio of correct estimations with adversarial samples. In some examples, the outcome of the ML trustworthiness assessment may comprise any combination of the outcomes above.

[0105] Then the NW 304 or the UE 302 may take further actions depending on the outcome, such as taking LCM decisions, e.g., one or more of activate, deactivate, switch, fallback AI / ML model and / or functionality, and inform each other. Specifically, in some examples, the LCM decision may be determined by the NW 304. As shown at 313a, the NW 304 determine the LCM decision based on the ML TW assessment outcome (e.g. model / functionality switching). Then the NW 304 may transmit, to the UE1 302, an indication or a configuration (the indication may be referred to as the second indication in the process 200) of the LCM decision based on the at least one outcome of the ML TW assessment. For example, the NW 304 may inform, to the UE1 302, the LCM decision based on the ML TW assessment outcome, as shown at 315a.

[0106] In some other examples, the LCM decision may be determined by the UE1 302. As shown at 313b, the UE1 302 determines the LCM decision based on the ML TW assessment outcome (e.g. model / functionality switching). Then the UE1 302 may transmit, to the NW 304, an indication or a configuration (the indication may be referred to as the third indication in the process 200) of the LCM decision based on the at least one outcome of the ML TW assessment. For example, the UE1 302 may inform, to the NW 304, the LCM decision basedon the ML TW assessment outcome, as shown at 315b.

[0107] FIG. 4 illustrates another example process 400 between a terminal device and a network device according to some embodiments of the present disclosure. Examples of the terminal device 402 may be a UE. Examples of the network device 404 may be an access network device, e.g. a gNodeB (gNB), or a core network (CN) device, e.g. a location management function (LMF) or an access and mobility management function (AMF), or other entities e.g., Administration and Maintenance (0AM), etc.

[0108] In the process 400, the network device 404 transmits (410), to the terminal device 402, an indication 405 (or referred to as a first indication 405) for requesting data 415 for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication 405 indicates trustworthiness related characteristics. In some examples, the at least one model or the at least one functionality is deployed at the network device 404. In some examples, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0109] The terminal device 402 receives (420), from the network device 404, the indication 405 for requesting data 415 for the trustworthiness assessment associated with the at least one model or the at least one functionality. The terminal device 402 transmits (430), to the network device 404, the data 415 with the trustworthiness related characteristics. On the network device 404 side, the network device 404 receives (440), from the terminal device 402, the data 415 with the trustworthiness related characteristics. Then the network device 404 may perform (450) the trustworthiness assessment based on the data.

[0110] In some examples, the trustworthiness related characteristics may comprise whether the data is to be used for the trustworthiness assessment. Alternatively or additionally, the trustworthiness related characteristics may comprise a number or a ratio of data samples for the trustworthiness assessment. Alternatively or additionally, the trustworthiness related characteristics may comprise a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment. In some examples, the trustworthiness assessment comprises explainability assessment, fairness assessment, or adversarial robustness assessment, or any combination thereof. In some examples, the data 415 may comprise at least one measurement of at least one reference signal. Alternatively or additionally, the data 415 may comprise at least one ground truth label or reward associated with the at least one measurement.

[0111] In some examples, the network device 404 may transmit, to the terminal device 402, at least one outcome of the trustworthiness assessment. Then the terminal device 402 may receive, from the network device 404, the at least one outcome of the trustworthiness assessment. In some examples, the at least one outcome of the trustworthiness assessment comprises at least one outcome of explainability assessment, at least one outcome of fairness assessment, or at least one outcome of adversarial robustness assessment, or any combination thereof.

[0112] In some examples, the network device 404 may determine, based on the at least one outcome of the trustworthiness assessment, a life cycle management (LCM) decision, in which the LCM decision is associated with the at least one model or the at least one functionality. Then the network device 404 may transmit, to the terminal device 402, an indication (may be referred to as a second indication) of the LCM decision based on the at least one outcome of the trustworthiness assessment. The terminal device 402 may receive, from the network device 404, the second indication of the LCM decision.

[0113] In some examples, the operations 410 to 440 above may also be suitable for some examples in which the at least one model or the at least one functionality is deployed at a terminal device, similar to a procedure of the network device 204 obtaining the data 215 from a terminal device (referred to as a second terminal device in the process 200) as described in the process 200 above. The details of such examples, may further refer to operations 303b and 305b in the FIG. 3. In such examples, in the operations 410 to 440, the terminal device 402 which provides the data 415 corresponds to the second terminal device in the process 200, and a terminal device being deployed at least one model or the at least one functionality corresponds to the terminal device 202 in the process 200.

[0114] According to the process 400, in some examples, the proposed method for data collection may be applied to the NW-side models, i.e., where the AI / ML model is deployed at a NW entity, e.g., a gNB or an LMF. The UE is employed by the NW to collect necessary data for assessing the AI / ML trustworthiness. In such examples, the operations in the process 400 are similar to those in the process 200 (in which the AI / ML model is deployed at a terminal device, e.g. the UE1 302 above), with roles of the UE and the NW exchanged in both process. Details of the process 400 may further refer to an example process 500 as illustrated in FIG. 5.

[0115] FIG. 5 illustrates another example process 500 according to some embodiments ofthe present disclosure. In the process 500, a UE 502 and a network device (referred to as NW) 504 are involved. The UE 502 may be an example of the terminal device 402. The NW 504 may be an example of the network device 404.

[0116] The NW 504 may request the data for trustworthiness (TW) assessment associated with at least one model (e.g. AI / ML model) or at least one functionality (e.g. AI / ML-enabled functionality or feature). For example, as shown at 501, the NW 504 requests measurements and / or ground truth labels by indicating certain characteristics. In other words, the NW 504 transmit, to the UE 502, an indication for requesting data for the trustworthiness assessment associated with the at least one model or the at least one functionality, in which the indication indicates certain characteristics (may be referred to as the trustworthiness related characteristics). In some examples, the requested data may comprise rewards instead of the ground truth labels.

[0117] At 503, the UE 502 provides, to the NW 504, the requested data with the indicated certain characteristics. In some examples, an example of the TW assessment may be ML TW trustworthiness assessment. Then the NW 504 may perform the ML TW (trustworthiness) assessment based on the data. For examples, the NW 504 performs the ML TW trustworthiness assessment based on the received data and configuration.

[0118] At 509, the NW 504 may determine the LCM decision based on the ML TW assessment outcome (e.g. model / functionality switching).

[0119] In some examples, as shown at 507, the NW 504 may optionally report its assessment outcome to the UE 502. Alternatively or additionally, as shown at 511, the NW 504 may optionally report or inform related LCM decision based on the ML TW assessment outcome to the UE 502, e.g., to provide more transparency, as shown in operation 511 in FIG. 5.

[0120] Depending on the AI / ML use case, as per the network node, a CN entity (e.g., LMF, AMF), a RAN node, e.g., gNB, or Operations, Administration and Maintenance (0AM) may perform one or more of the operations performed by the NW 304 or 504 described above. Depending on the AI / ML use case, signaling between the NW 504 and the UE 502 (or NW 304 and the UE 302) for the above procedures may take place via different protocols, such as (equivalent of) LTE positioning protocol (LPP) (between the LMF and the UE), radio resource control (RRC) (between the gNB and the UE), media access control (MAC) (between the gNB and the UE), etc. in the fifth generation (5G).

[0121] Depending on the AI / ML use case, measurements in the data samples (i.e. the data) may relate to different reference signals, e.g., reference signals for positioning, channel sounding or channel state information, synchronization, phase tracking, demodulation, etc., as (equivalent of) Positioning Reference Signal (PRS) / Sounding Reference Signal (SRS), Channel State Information Reference Signal (CSI-RS), Demodulation Reference Signal (DMRS), Phase Tracking Reference Signal (PTRS), primary synchronization signal (PSS) / Secondary synchronization signal (SSS) in 5G new radio (NR).

[0122] In some embodiments of the present disclosure, incorporating trustworthiness aspect of AI / ML into data collection procedures at RAN would enhance the performance of AI / ML use cases with regards to their explainability, fairness, and robustness, as well as enable efficient usage of radio resources by avoiding unnecessary signaling and processing overhead for data collection, since the proposed method enables collection of only those data with certain AI / ML trustworthy-related characteristics that are suitable and useful for assessing AI / ML trustworthiness. Some examples are related to trustworthiness aspects of AI / ML, and may be applied in sixth generation (6G) use cases.

[0123] FIG. 6 illustrates a flowchart of an example method 600 implemented at a terminal device in accordance with some embodiments of the present disclosure. An example of the terminal device may be the terminal device 202.

[0124] At block 610, the terminal device transmits, to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics. At block 620, the terminal device receives, from the network device, the data with the trustworthiness related characteristics. At block 630, the terminal device performs the trustworthiness assessment based on the data.

[0125] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0126] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0127] In some embodiments, the terminal device may receive, from the network device, configuration information for reporting at least one outcome of the trustworthiness assessment.

[0128] In some embodiments, the trustworthiness assessment is performed further based on the configuration information.

[0129] In some embodiments, the terminal device may report, to the network device, at least one outcome of the trustworthiness assessment.

[0130] In some embodiments, the indication is a first indication, and the terminal device may receive, from the network device, a second indication of a life cycle management (LCM) decision based on at least one outcome of the trustworthiness assessment, in which the LCM decision is associated with the at least one model or the at least one functionality.

[0131] In some embodiments, the indication is a first indication, and the terminal device may determine, based on at least one outcome of the trustworthiness assessment, an LCM decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and transmit, to the network device, a third indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

[0132] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0133] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0134] In some embodiments, the trustworthiness assessment is performed based on one of the following: receiving, from the network device, an explicit request for performing the trustworthiness assessment; a first trigger configured by the network device for performing the trustworthiness assessment; or a second trigger predetermined by the terminal device for performing the trustworthiness assessment.

[0135] In some embodiments, the first trigger is associated with at least one of the following: a time interval for performing the trustworthiness assessment periodically; or criteria for triggering the trustworthiness assessment to be performed.

[0136] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; atleast one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0137] FIG. 7 illustrates a flowchart of an example method 700 implemented at a network device in accordance with some embodiments of the present disclosure. An example of the network device may be the network device 204.

[0138] At block 710, the network device receives, from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics. At block 720, the network device obtains the data with the trustworthiness related characteristics. At block 730, the network device transmits, to the terminal device, the data with the trustworthiness related characteristics.

[0139] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0140] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0141] In some embodiments, the network device is caused to obtain the data by: generating the data based with the trustworthiness related characteristics.

[0142] In some embodiments, the terminal device is a first terminal device, the indication is a first indication, and the network device is caused to obtain the data by: transmitting, to a second terminal device, a fourth indication for requesting the data, wherein the fourth indication indicates the trustworthiness related characteristics; and receiving, from the second terminal device, the data with the trustworthiness related characteristics.

[0143] In some embodiments, the network device may transmit, to the terminal device, configuration information for reporting at least one outcome of the trustworthiness assessment.

[0144] In some embodiments, the network device may receive, from the terminal device, at least one outcome of the trustworthiness assessment.

[0145] In some embodiments, the indication is a first indication, and the network device may determine, based on at least one outcome of the trustworthiness assessment, a life cycle management (LCM) decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and transmit, to the terminal device, a second indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

[0146] In some embodiments, the indication is a first indication, and the network device may receive, from the terminal device, a third indication of an LCM decision based on at least one outcome of the trustworthiness assessment, in which the LCM decision is associated with the at least one model or the at least one functionality.

[0147] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0148] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0149] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0150] It should be noted that those embodiments described with reference FIG. 6 also apply for or could be combined with the embodiments described with reference FIG. 7, which are omitted here for brevity.

[0151] In some embodiments, an apparatus (for example, the terminal device 202) capable of performing the method 600 may comprise means for performing the respective steps of the method 600. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module.

[0152] In some embodiments, the apparatus may comprise means for transmitting, to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; means for receiving, from the network device, the data with the trustworthiness related characteristics; and means for performing the trustworthiness assessment based on the data.

[0153] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0154] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0155] In some embodiments, the apparatus may comprise means for receiving, from the network device, configuration information for reporting at least one outcome of the trustworthiness assessment.

[0156] In some embodiments, the trustworthiness assessment is performed further based on the configuration information.

[0157] In some embodiments, the apparatus may comprise means for reporting, to the network device, at least one outcome of the trustworthiness assessment.

[0158] In some embodiments, the indication is a first indication, and the apparatus may comprise means for receiving, from the network device, a second indication of a life cycle management (LCM) decision based on at least one outcome of the trustworthiness assessment, in which the LCM decision is associated with the at least one model or the at least one functionality.

[0159] In some embodiments, the indication is a first indication, and the apparatus may comprise means for determining, based on at least one outcome of the trustworthiness assessment, an LCM decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and means for transmitting, to the network device, a third indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

[0160] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0161] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0162] In some embodiments, the trustworthiness assessment is performed based on one of the following: receiving, from the network device, an explicit request for performing the trustworthiness assessment; a first trigger configured by the network device for performing the trustworthiness assessment; or a second trigger predetermined by the terminal device for performing the trustworthiness assessment.

[0163] In some embodiments, the first trigger is associated with at least one of the following: a time interval for performing the trustworthiness assessment periodically; or criteria for triggering the trustworthiness assessment to be performed.

[0164] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0165] In some embodiments, the apparatus further comprises means for performing other steps in some embodiments of the method 600. In some embodiments, the means comprises at least one processor; and at least one memory including computer program code, the at least one memory and computer program code configured to, with the at least one processor, cause the performance of the apparatus.

[0166] In some embodiments, an apparatus (for example, the network device 204) capable of performing the method 700 may comprise means for performing the respective steps of the method 700. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module.

[0167] In some embodiments, the apparatus may comprise means for receiving, from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; means for obtaining the data with the trustworthiness related characteristics; and means for transmitting, to the terminal device, the data with the trustworthiness related characteristics.

[0168] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0169] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0170] In some embodiments, the means for obtaining the data may comprise means for generating the data based with the trustworthiness related characteristics.

[0171] In some embodiments, the terminal device is a first terminal device, the indication is a first indication, and the means for obtaining the data may comprise means for transmitting, to a second terminal device, a fourth indication for requesting the data, wherein the fourth indication indicates the trustworthiness related characteristics; and means for receiving, from the second terminal device, the data with the trustworthiness related characteristics.

[0172] In some embodiments, the apparatus may comprise means for transmitting, to the terminal device, configuration information for reporting at least one outcome of the trustworthiness assessment.

[0173] In some embodiments, the apparatus may comprise means for receiving, from the terminal device, at least one outcome of the trustworthiness assessment.

[0174] In some embodiments, the indication is a first indication, and the apparatus may comprise means for determining, based on at least one outcome of the trustworthiness assessment, a life cycle management (LCM) decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and means for transmitting, to the terminal device, a second indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

[0175] In some embodiments, the indication is a first indication, and the apparatus may comprise means for receiving, from the terminal device, a third indication of an LCM decision based on at least one outcome of the trustworthiness assessment, in which the LCM decision is associated with the at least one model or the at least one functionality.

[0176] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0177] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0178] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0179] In some embodiments, the apparatus further comprises means for performing other steps in some embodiments of the method 700. In some embodiments, the means comprises at least one processor; and at least one memory including computer program code, the at least one memory and computer program code configured to, with the at least one processor, cause the performance of the apparatus.

[0180] FIG. 8 illustrates a flowchart of an example method 800 implemented at a terminal device in accordance with some other embodiments of the present disclosure. An example of the terminal device may be the terminal device 402.

[0181] At block 810, the terminal device receives, from a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics. At block 820, the terminal device transmits, to the network device, the data with the trustworthiness related characteristics.

[0182] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0183] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0184] In some embodiments, the terminal device may receive, from the network device, at least one outcome of the trustworthiness assessment.

[0185] In some embodiments, the indication is a first indication, and the terminal device may receive, from the network device, a second indication of a life cycle management (LCM) decision based on at least one outcome of the trustworthiness assessment, in which the LCM decision is associated with the at least one model or the at least one functionality.

[0186] In some embodiments, the at least one model or the at least one functionality is deployed at the network device.

[0187] In some embodiments, the terminal device is a first terminal device, and the at least one model or the at least one functionality is deployed at a second terminal device.

[0188] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0189] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0190] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0191] FIG. 9 illustrates a flowchart of an example method implemented at a network device in accordance with some other embodiments of the present disclosure. An example of the network device may be the network device 404.

[0192] At block 910, the network device transmits, to a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics. At block 920, the network device receives, from the terminal device, the data with the trustworthiness related characteristics. At block 930, the network device performs the trustworthiness assessment based on the data.

[0193] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0194] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0195] In some embodiments, the network device may transmit, to the terminal device, at least one outcome of the trustworthiness assessment.

[0196] In some embodiments, the indication is a first indication, and the network device may determine, based on at least one outcome of the trustworthiness assessment, a life cycle management (LCM) decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and transmits, to the terminal device, a second indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

[0197] In some embodiments, the at least one model or the at least one functionality is deployed at the network device.

[0198] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0199] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0200] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0201] Those embodiments described with reference FIG. 8 also apply for or could be combined with the embodiments described with reference FIG. 9, which are omitted here for brevity.

[0202] In some embodiments, an apparatus (for example, the terminal device 402) capable of performing the method 800 may comprise means for performing the respective steps of the method 800. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module.

[0203] In some embodiments, the apparatus may comprise means for receiving, from a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, in which the indication indicates trustworthiness related characteristics; and means for transmitting, to the network device, the data with the trustworthiness related characteristics.

[0204] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; anumber or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0205] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0206] In some embodiments, the apparatus may comprise means for receiving, from the network device, at least one outcome of the trustworthiness assessment.

[0207] In some embodiments, the indication is a first indication, and the apparatus may comprise means for receiving, from the network device, a second indication of a life cycle management (LCM) decision based on at least one outcome of the trustworthiness assessment, in which the LCM decision is associated with the at least one model or the at least one functionality.

[0208] In some embodiments, the at least one model or the at least one functionality is deployed at the network device.

[0209] In some embodiments, the terminal device is a first terminal device, and the at least one model or the at least one functionality is deployed at a second terminal device.

[0210] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0211] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0212] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0213] In some embodiments, the apparatus further comprises means for performing other steps in some embodiments of the method 800. In some embodiments, the means comprises at least one processor and at least one memory including computer program code, the at least one memory and computer program code configured to, with the at least one processor, cause the performance of the apparatus.

[0214] In some embodiments, an apparatus (for example, the network device 404) capable of performing the method 900 may comprise means for performing the respective steps of the method 900. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module.

[0215] In some embodiments, the apparatus may comprise means for transmitting, to a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; means for receiving, from the terminal device, the data with the trustworthiness related characteristics; and means for performing the trustworthiness assessment based on the data.

[0216] In some embodiments, the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

[0217] In some embodiments, the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

[0218] In some embodiments, the apparatus may comprise means for transmitting, to the terminal device, at least one outcome of the trustworthiness assessment.

[0219] In some embodiments, the indication is a first indication, and the apparatus may comprise means for determining, based on at least one outcome of the trustworthiness assessment, a life cycle management (LCM) decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and means for transmitting, to the terminal device, a second indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

[0220] In some embodiments, the at least one model or the at least one functionality is deployed at the network device.

[0221] In some embodiments, the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

[0222] In some embodiments, the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

[0223] In some embodiments, the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

[0224] In some embodiments, the apparatus further comprises means for performing other steps in some embodiments of the method 900. In some embodiments, the means comprises at least one processor and at least one memory including computer program code, the at least one memory and computer program code configured to, with the at least one processor, cause the performance of the apparatus.

[0225] FIG. 10 illustrates an example simplified block diagram of a device 1000 that is suitable for implementing embodiments of the present disclosure. The device 1000 may be provided to implement a communication device or a network element, for example, the terminal device 202 or 402, or the network device 204 or 404 as shown in FIG. 2 or FIG. 4. As shown, the device 1000 includes one or more processors 1010, one or more memories 1020 may couple to the processor 1010, and one or more communication modules 1040 may couple to the processor 1010.

[0226] The communication module 1040 is for bidirectional communications. The communication module 1040 has at least one antenna to facilitate communication. The communication interface may represent any interface that is necessary for communication with other network elements, for example the communication interface may be wireless or wireline to other network elements, or software based interface for communication.

[0227] The processor 1010 may be of any type suitable to the local technical network and may include one or more of the following: general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multicore processor architecture, as non-limiting examples. The device 1000 may have multiple processors, such as an application specific integrated circuit chip that is slaved in time to a clock which synchronizes the main processor.

[0228] The memory 1020 may include one or more non-volatile memories and one or more volatile memories. Examples of the non-volatile memories include, but are not limited to, aread only memory (ROM) 1024, an electrically programmable read only memory (EPROM), a flash memory, a hard disk, a compact disc (CD), a digital video disk (DVD), and other magnetic storage and / or optical storage. Examples of the volatile memories include, but are not limited to, a random access memory (RAM) 1022 and other volatile memories that will not last in the power-down duration. The memory 1020 may store instructions that, when executed by the processor 1010, cause the apparatus 1000 to perform any of the methods as disclosed herein.

[0229] A computer program 1030 includes computer executable instructions that are executed by the associated processor 1010. The program 1030 may be stored in the ROM 1024. The processor 1010 may perform any suitable actions and processing by loading the program 1030 into the RAM 1022.

[0230] The embodiments of the present disclosure may be implemented by means of the program so that the device 1000 may perform any process of the disclosure as discussed with reference to FIG. 1 to FIG. 9. The embodiments of the present disclosure may also be implemented by hardware or by a combination of software and hardware.

[0231] In some embodiments, the program 1030 may be tangibly contained in a computer readable medium which may be included in the device 1000 (such as in the memory 1020) or other storage devices that are accessible by the device 1000. The device 1000 may load the program 1030 from the computer readable medium to the RAM 1022 for execution. The computer readable medium may include any types of tangible non-volatile storage, such as ROM, EPROM, a flash memory, a hard disk, CD, DVD, and the like. FIG. 11 shows an example of the computer readable medium 1100 in form of CD or DVD. The computer readable medium has the program 1030 stored thereon.

[0232] Generally, various embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device. While various aspects of embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representations, it is to be understood that the block, apparatus, system, technique or method described herein may be implemented in, as non-limiting examples, hardware, software,firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0233] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer readable storage medium. The computer program product includes computer-executable instructions, such as those included in program modules, being executed in a device on a target real or virtual processor, to carry out the methods 200 to 900 as described above with reference to FIG. 2 to FIG. 9. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, or the like that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Machine-executable instructions for program modules may be executed within a local or distributed device. In a distributed device, program modules may be located in both local and remote storage media.

[0234] Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0235] In the context of the present disclosure, the computer program codes or related data may be carried by any suitable carrier to enable the device, apparatus or processor to perform various processes and operations as described above. Examples of the carrier include a signal, computer readable medium, and the like.

[0236] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory(EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD- ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0237] Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0238] Although the present disclosure has been described in languages specific to structural features and / or methodological acts, it is to be understood that the present disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

WHAT IS CLAIMED IS:

1. A terminal device comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the terminal device at least to: transmit, to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; receive, from the network device, the data with the trustworthiness related characteristics; and perform the trustworthiness assessment based on the data.

2. The terminal device of claim 1, wherein the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

3. The terminal device of claim 1 or 2, wherein the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or adversarial robustness assessment.

4. The terminal device of any of claims 1-3, wherein the terminal device is further caused to: receive, from the network device, configuration information for reporting at least one outcome of the trustworthiness assessment.

5. The terminal device of claim 4, wherein the trustworthiness assessment is performed further based on the configuration information.

456. The terminal device of any of claims 1-5, wherein the terminal device is further caused to: report, to the network device, at least one outcome of the trustworthiness assessment.

7. The terminal device of any of claims 1-6, wherein the indication is a first indication, and the terminal device is further caused to: receive, from the network device, a second indication of a life cycle management (LCM) decision based on at least one outcome of the trustworthiness assessment, wherein the LCM decision is associated with the at least one model or the at least one functionality.

8. The terminal device of any of claims 1-6, wherein the indication is a first indication, and the terminal device is further caused to: determine, based on at least one outcome of the trustworthiness assessment, an LCM decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and transmit, to the network device, a third indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

9. The terminal device of any of claims 1-8, wherein the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

10. The terminal device of any of claims 1-9, wherein the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

11. The terminal device of any of claims 1-10, wherein the trustworthiness assessment is performed based on one of the following: receiving, from the network device, an explicit request for performing the trustworthiness assessment; a first trigger configured by the network device for performing the trustworthiness assessment; or a second trigger predetermined by the terminal device for performing the trustworthiness assessment.4612. The terminal device of claim 11, wherein the first trigger is associated with at least one of the following: a time interval for performing the trustworthiness assessment periodically; or criteria for triggering the trustworthiness assessment to be performed.

13. The terminal device of any of claims 4 and 6-8, wherein the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

14. A network device comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the network device at least to: receive, from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; obtain the data with the trustworthiness related characteristics; and transmit, to the terminal device, the data with the trustworthiness related characteristics.

15. The network device of claim 14, wherein the trustworthiness related characteristics comprise at least one of the following: whether the data is to be used for the trustworthiness assessment; a number or a ratio of data samples for the trustworthiness assessment; or a number or a ratio of synthetic data samples for at least one part of the data for the trustworthiness assessment.

16. The network device of claim 14 or 15, wherein the trustworthiness assessment comprises at least one of the following: explainability assessment; fairness assessment; or47adversarial robustness assessment.

17. The network device of any of claims 14-16, wherein the network device is caused to obtain the data by: generating the data based with the trustworthiness related characteristics.

18. The network device of any of claims 14-16, wherein the terminal device is a first terminal device, the indication is a first indication, and the network device is caused to obtain the data by: transmitting, to a second terminal device, a fourth indication for requesting the data, wherein the fourth indication indicates the trustworthiness related characteristics; and receiving, from the second terminal device, the data with the trustworthiness related characteristics.

19. The network device of any of claims 14-18, wherein the network device is further caused to: transmit, to the terminal device, configuration information for reporting at least one outcome of the trustworthiness assessment.

20. The network device of any of claims 13-19, wherein the network device is further caused to: receive, from the terminal device, at least one outcome of the trustworthiness assessment.

21. The network device of any of claims 14-20, wherein the indication is a first indication, and the network device is further caused to: determine, based on at least one outcome of the trustworthiness assessment, a life cycle management (LCM) decision, wherein the LCM decision is associated with the at least one model or the at least one functionality; and transmit, to the terminal device, a second indication of the LCM decision based on the at least one outcome of the trustworthiness assessment.

22. The network device of any of claims 14-20, wherein the indication is a first indication, and the network device is further caused to:receive, from the terminal device, a third indication of an LCM decision based on at least one outcome of the trustworthiness assessment, wherein the LCM decision is associated with the at least one model or the at least one functionality.

23. The network device of any of claims 14-22, wherein the at least one model or the at least one functionality is based on artificial intelligence (Al) / machine learning (ML).

24. The network device of any of claims 14-23, wherein the data comprises at least one of the following: at least one measurement of at least one reference signal; or at least one ground truth label or reward associated with the at least one measurement.

25. The network device of any of claims 19-21, wherein the at least one outcome of the trustworthiness assessment comprises at least one of the following: at least one outcome of explainability assessment; at least one outcome of fairness assessment; or at least one outcome of adversarial robustness assessment.

26. A method comprising: transmitting, at a terminal device and to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; receiving, from the network device, the data with the trustworthiness related characteristics; and performing the trustworthiness assessment based on the data.

27. A method comprising: receiving, at a network device and from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; obtaining the data with the trustworthiness related characteristics; and transmitting, to the terminal device, the data with the trustworthiness related characteristics.

28. An apparatus comprising: means for transmitting, at a terminal device and to a network device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; means for receiving, from the network device, the data with the trustworthiness related characteristics; and means for performing the trustworthiness assessment based on the data.

29. An apparatus comprising: means for receiving, at a network device and from a terminal device, an indication for requesting data for trustworthiness assessment associated with at least one model or at least one functionality, wherein the indication indicates trustworthiness related characteristics; means for obtaining the data with the trustworthiness related characteristics; and means for transmitting, to the terminal device, the data with the trustworthiness related characteristics.

30. A computer readable medium comprising program instructions stored thereon for performing at least the method of claim 26 or 27.

Images