Model performance verification method, communication device, communication system and storage medium

Abstract

A model performance verification method, a communication device, a communication system and a storage medium. The method comprises: sending first information to a network device, wherein the first information is used for indicating that a terminal is configured with an executable first artificial intelligence (AI) model, and the first AI model is an AI model in a two-sided model that is applied to the terminal. According to the method, since first information is used for indicating that a terminal is configured with an executable first AI model, the terminal sending the first information to a network device enables the network device to determine that the first AI model in a two-sided model that is applied to the terminal is executable, thereby implementing performance verification on the first AI model by the terminal and the network device. Therefore, it is ensured that the terminal and the network device can normally use the first AI model, thereby improving the reliability of the first AI model in a two-sided model service scenario.

Description

Model performance verification methods, communication equipment, communication systems and storage media Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to a model performance verification method, communication equipment, communication system and storage medium. Background Technology

[0002] In wireless communication systems, artificial intelligence (AI) can be used for prediction and reasoning to improve system performance. Machine learning algorithms are currently one of the most important methods for implementing AI technology. Machine learning algorithms can obtain AI models through large amounts of training data, and these AI models can then be used to predict events. Summary of the Invention

[0003] To verify the performance of a bilateral model, embodiments of this disclosure propose a model performance verification method, a communication device, a communication system, and a storage medium.

[0004] According to a first aspect of the embodiments of this disclosure, a model performance verification method is proposed, executed by a terminal, the method comprising:

[0005] Send the first message to the network device;

[0006] The first information is used to indicate that the terminal is configured with an executable first artificial intelligence (AI) model; the first AI model is the AI ​​model applied to the terminal in the bilateral model.

[0007] According to a second aspect of the embodiments of this disclosure, a model performance verification method is proposed, performed by a network device, the method comprising:

[0008] The first message sent by the receiving terminal;

[0009] The first information is used to indicate that the terminal is configured with an executable first artificial intelligence (AI) model; the first AI model is the AI ​​model applied to the terminal in a two-sided model.

[0010] According to a third aspect of the embodiments of this disclosure, a communication device is provided that can be used to perform the methods described in an optional implementation of the first or second aspect.

[0011] According to a fourth aspect of the present disclosure, a communication system is proposed, including a terminal and a network device, wherein the terminal is configured to implement the model performance verification method described in the first aspect, and the network device is configured to implement the model performance verification method described in the second aspect.

[0012] According to a fifth aspect of the present disclosure, a storage medium is provided that stores instructions that, when executed on a communication device, cause the communication device to perform the model performance verification method as described in the first or second aspect.

[0013] According to a sixth aspect of the present disclosure, a program product is proposed, including at least one of a program and instructions, wherein when the program and instructions are executed by a communication device, the model performance verification method described in the first or second aspect is implemented.

[0014] By adopting the above technical solution, at least the following beneficial technical effects can be achieved:

[0015] Since the first information is used to indicate that an executable first AI model is configured on the terminal, the terminal sends the first information to the network device, enabling the network device to determine that the first AI model applied to the terminal in the bilateral model is executable. This realizes the performance verification of the first AI model by the terminal and the network device, thereby ensuring that the terminal and the network device can use the first AI model normally and improving the reliability of the first AI model in the bilateral model business scenario.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings required for the description of the embodiments are introduced below. The following drawings are only some embodiments of this disclosure and do not impose specific limitations on the protection scope of this disclosure.

[0018] Figure 1 is a schematic diagram of the architecture of a communication system according to an embodiment of the present disclosure.

[0019] Figure 2A is an interactive schematic diagram of a model performance verification method according to an embodiment of the present disclosure.

[0020] Figure 2B is an interactive schematic diagram of a model performance verification method according to an embodiment of the present disclosure.

[0021] Figure 2C is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure.

[0022] Figure 2D is an interactive schematic diagram of a model performance verification method according to an embodiment of the present disclosure.

[0023] Figure 3A is a flowchart illustrating a model performance verification method according to an embodiment of the present disclosure.

[0024] Figure 3B is a flowchart illustrating a model performance verification method according to an embodiment of the present disclosure.

[0025] Figure 4A is a schematic flowchart illustrating a model performance verification method according to an embodiment of the present disclosure.

[0026] Figure 4B is a flowchart illustrating a model performance verification method according to an embodiment of the present disclosure.

[0027] Figure 5A is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure.

[0028] Figure 5B is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure.

[0029] Figure 6A is a schematic diagram of the structure of a terminal proposed in an embodiment of this disclosure.

[0030] Figure 6B is a schematic diagram of the structure of a network device proposed in an embodiment of this disclosure.

[0031] Figure 7A is a schematic diagram of the structure of the communication device proposed in an embodiment of this disclosure.

[0032] Figure 7B is a schematic diagram of the chip structure proposed in an embodiment of this disclosure. Detailed Implementation

[0033] This disclosure presents a model performance verification method, a communication device, a communication system, and a storage medium.

[0034] In a first aspect, embodiments of this disclosure propose a model performance verification method, executed by a terminal, the method comprising:

[0035] Send the first message to the network device;

[0036] The first information is used to indicate that the terminal is configured with an executable first artificial intelligence (AI) model; the first AI model is the AI ​​model applied to the terminal in the bilateral model.

[0037] In the above embodiments, since the first information is used to indicate that an executable first AI model is configured on the terminal, the terminal sends the first information to the network device, so that the network device determines that the first AI model applied to the terminal in the bilateral model is executable. This realizes the performance verification of the first AI model by the terminal and the network device, thereby ensuring that the terminal and the network device can use the first AI model normally and improving the reliability of the first AI model in the bilateral model business scenario.

[0038] In conjunction with some embodiments of the first aspect, in some embodiments, sending the first information to the network device includes:

[0039] Once the performance of the first AI model meets the performance requirements, the first information is sent to the network device; the performance requirements are used to indicate the conditions required for the model performance to reach the expected performance.

[0040] In the above embodiments, the terminal sends the first information to the network device only when the performance of the first AI model meets the performance requirements. For example, the terminal or its over-the-top (OTT) server trains the first AI model, and the terminal sends the first information to the network device when the trained first AI model meets the performance requirements. Through the above method, the situation where the first AI model is still unavailable after the terminal sends the first information to the network device is avoided, and the reliability of the first AI model in the two-sided model service scenario is further improved.

[0041] In conjunction with some embodiments of the first aspect, in some embodiments, before sending the first information to the network device, the method further includes:

[0042] Obtain the performance requirement information of the first AI model; the performance requirement information is used to evaluate whether the model performance of the first AI model meets the performance requirements, and the performance requirements are used to indicate the required conditions for the model performance to reach the expected performance.

[0043] Based on the performance requirements information, the performance of the first AI model is evaluated.

[0044] In the above embodiments, before the terminal sends the first information to the network device, the performance of the first AI model is evaluated based on the obtained performance requirement information. The terminal determines whether the trained first AI model meets the performance requirements based on the model performance evaluation results, ensuring that the first information is sent to the network device when the first AI model meets the performance requirements and is executable, thus guaranteeing the availability of the first AI model.

[0045] In conjunction with some embodiments of the first aspect, in some embodiments, obtaining the performance requirement information of the first AI model includes any one of the following:

[0046] Receive the performance requirement information sent by the network device;

[0047] The OTT server sends the performance requirement information; the performance requirement information is sent to the terminal by the OTT server after receiving the performance requirement information sent by the network device.

[0048] In the above embodiments, on the one hand, the terminal can directly receive the performance requirement information sent by the network device. By using the above method, the terminal and the network device can directly transmit data, which can reduce information transmission latency, reduce the risk of data packet loss, and ensure low latency and data reliability for the terminal to receive performance requirement information.

[0049] On the other hand, the terminal can receive performance requirement information from network devices sent by the terminal's OTT server. In the above method, the OTT server can integrate data from different network devices, such as data from network devices of different operators. The terminal can directly receive performance requirement information from different network devices through the OTT server, so that the terminal does not need to be adapted to different network device operators. The terminal can be compatible with the performance requirement information sent by different network device operators, which improves the efficiency of the terminal in receiving performance requirement information.

[0050] In conjunction with some embodiments of the first aspect, in some embodiments, the performance requirement information includes at least one of model validation requirement information and model accuracy requirement information; the model validation requirement information includes relevant information for validating the performance of the first AI model; the model accuracy requirement information is used to indicate the accuracy requirement conditions when the model performance reaches the expected performance.

[0051] The evaluation of the AI ​​model's performance based on the performance requirement information includes at least one of the following:

[0052] Based on the model verification requirements, the performance of the first AI model is verified;

[0053] Based on the model accuracy requirement information, the accuracy of the first AI model is calibrated.

[0054] In the above embodiments, on the one hand, the terminal verifies the performance of the first AI model based on the model verification requirement information, such as verifying whether the model parameters of the first AI model are aligned between the terminal and the network device, verifying whether the functions of the first AI model are complete, and verifying whether the terminal and the network device are compatible with the first AI model, so as to ensure the consistency and compatibility of the first AI model on the terminal and the network device, thereby ensuring the availability of the first AI model.

[0055] On the other hand, the terminal calibrates the accuracy of the first AI model based on the model accuracy requirements, ensuring that the accuracy of the first AI model meets the requirements and improving the accuracy of the first AI model.

[0056] On the other hand, the terminal verifies the performance of the first AI model to ensure its consistency, functional integrity, and compatibility; it calibrates the accuracy of the first AI model to optimize it, thus ensuring its performance and accuracy; through the above methods, efficient collaboration between the terminal and network devices on the first AI model is achieved.

[0057] In conjunction with some embodiments of the first aspect, in some embodiments, the first AI model is used to compress Channel State Information (CSI).

[0058] In the above embodiments, the CSI compression model is applied to the bilateral model scenario. The terminal compresses the CSI through the first AI model and works in conjunction with the network device to complete the compression and decoding of the CSI, which can greatly reduce the amount of CSI data and thus reduce the bandwidth occupation of uplink transmission.

[0059] In conjunction with some embodiments of the first aspect, in some embodiments, verifying the performance of the first AI model based on the model validation requirement information includes at least one of the following:

[0060] The performance of the first AI model is verified based on the first CSI and the first CSI feedback.

[0061] Based on the first CSI and the first model information, the performance of the first AI model is verified; the first model information includes at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model trained by the network device, and the model structure of the first AI model trained by the network device.

[0062] Based on the first CSI and the second model information, the performance of the first AI model is verified; the second model information includes at least one of the following: the second AI model, the model parameters of the second AI model, and the model structure of the second AI model; the second AI model is the AI ​​model applied to the network device in the bilateral model;

[0063] Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

[0064] In the above embodiments, the performance of the first AI model was verified through different methods, ensuring its usability. For example, the first CSI was used as an ideal benchmark CSI, and the first CSI feedback included the ideal compression result of the first AI model on the first CSI. Based on the first CSI and the first CSI feedback, the terminal effectively verified the functionality and CSI compression performance of the first AI model.

[0065] For example, based on the first model information, the terminal can obtain at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model, and the model structure of the first AI model. Based on the first CSI and the first model information, the terminal can effectively verify whether the parameters, functions, and performance of the first AI model are aligned between the terminal and the network device.

[0066] For example, based on the second model information, the terminal can obtain at least one of the following: the second AI model applied to the network device, the model parameters of the second AI model, and the model structure of the second AI model. Based on the first CSI and the second model information, the terminal can effectively verify whether the result of the terminal compressing the first CSI using the first AI model can be successfully restored through the second AI model. This achieves closed-loop verification of CSI compression and restoration in collaboration between the terminal and the network device, ensuring the compatibility of the terminal and the network device with the first AI model and the availability of the first AI model.

[0067] In conjunction with some embodiments of the first aspect, in some embodiments, the model accuracy requirement information includes at least one of the following:

[0068] The first model accuracy requirement information is used to require that the accuracy of the first AI model after training is greater than a first threshold.

[0069] The second model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than a second threshold.

[0070] The third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the third threshold.

[0071] The fourth model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the fourth threshold and greater than the fifth threshold.

[0072] The fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the sixth threshold and greater than the seventh threshold.

[0073] The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the eighth threshold.

[0074] The seventh model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the ninth threshold;

[0075] The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold.

[0076] The ninth model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the twelfth threshold and greater than the thirteenth threshold;

[0077] Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

[0078] In the above embodiments, the terminal calibrates the accuracy of the first AI model based on the model accuracy requirement information to ensure that the accuracy of the first AI model meets the requirements and improves the accuracy of the first AI model.

[0079] In conjunction with some embodiments of the first aspect, in some embodiments, the performance requirement information is carried in any of the following:

[0080] MAC Control Element (MAC CE);

[0081] Physical layer signaling;

[0082] Radio Resource Control (RRC);

[0083] User interface;

[0084] The interface between the terminal and the network node used for AI control.

[0085] In the above embodiments, the performance requirement information is carried in any of the above items without additional signaling overhead, thus saving system resources.

[0086] In conjunction with some embodiments of the first aspect, in some embodiments, the first information is sent to the network device via any of the following:

[0087] MAC CE;

[0088] Physical layer signaling;

[0089] RRC;

[0090] User interface;

[0091] The interface between the terminal and the network node used for AI control;

[0092] Update the applicable function report; the updated applicable function report is obtained by updating the applicable function report, which is sent to the network device after the terminal receives the inference configuration sent by the network device; the inference configuration is sent to the terminal by the network device after the terminal receives the terminal capability query sent by the network device and sends the terminal capability report to the network device.

[0093] In the above embodiments, the terminal sends the first information to the network device through any of the above methods without additional signaling overhead, thus saving system resources.

[0094] In conjunction with some embodiments of the first aspect, in some embodiments, the first AI model is obtained through any of the following methods:

[0095] The first AI model is trained based on the model data of the first AI model;

[0096] The first AI model is received from the OTT server; the first AI model is obtained by the OTT server after receiving the model data of the first AI model sent by the network device, and training it based on the model data.

[0097] The model data includes at least one of the training data and model parameters of the first AI model.

[0098] In the above embodiments, on the one hand, the terminal can train the first AI model locally based on the model data of the first AI model, which helps to avoid the leakage of sensitive data such as model data and ensure data security; on the other hand, the terminal can continuously optimize the first AI model based on the model data to achieve model optimization on the terminal locally.

[0099] On the other hand, the terminal can receive the first AI model sent by the OTT server. That is, the OTT server trains the first AI model based on the model data, so that the terminal does not need to train the first AI model, saving the terminal's computing resources and reducing the terminal's energy consumption.

[0100] Secondly, this disclosure provides a model performance verification method, executed by a network device, the method comprising:

[0101] The first message sent by the receiving terminal;

[0102] The first information is used to indicate that the terminal is configured with an executable first artificial intelligence (AI) model; the first AI model is the AI ​​model applied to the terminal in a two-sided model.

[0103] In conjunction with some embodiments of the second aspect, in some embodiments, before the first information sent by the receiving terminal, the method further includes any one of the following:

[0104] Send performance requirement information to the terminal;

[0105] Send performance requirement information to the OTT server;

[0106] The performance requirement information is used to evaluate whether the performance of the first AI model meets the performance requirements, and the performance requirements are used to indicate the conditions required for the model performance to reach the expected performance.

[0107] In conjunction with some embodiments of the second aspect, in some embodiments, the performance requirement information is carried in any of the following:

[0108] MAC control element MAC CE;

[0109] Physical layer signaling;

[0110] Radio Resource Control (RRC);

[0111] User interface;

[0112] The interface between the terminal and the network node used for AI control.

[0113] In conjunction with some embodiments of the second aspect, in some embodiments, the method further includes:

[0114] Based on the performance requirements of the first AI model, the model performance of the first AI model is evaluated.

[0115] The performance requirement information is used to evaluate whether the performance of the first AI model meets the performance requirements, and the performance requirements are used to indicate the conditions required for the model performance to reach the expected performance.

[0116] In conjunction with some embodiments of the second aspect, in some embodiments, the performance requirement information includes at least one of the following:

[0117] Model validation requirements information, including relevant information for validating the performance of the first AI model;

[0118] Model accuracy requirement information indicates the accuracy requirements under which the model's performance reaches the desired performance.

[0119] In conjunction with some embodiments of the second aspect, in some embodiments, the first AI model is used to compress the channel state information (CSI).

[0120] In conjunction with some embodiments of the second aspect, in some embodiments, the model validation requirement information includes at least one of the following:

[0121] First CSI and First CSI Feedback;

[0122] First CSI and first model information; the first model information includes at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model trained by the network device, and the model structure of the first AI model trained by the network device;

[0123] First CSI and second model information; the second model information includes at least one of the following: second AI model, model parameters of the second AI model, and model structure of the second AI model; the second AI model is the AI ​​model applied to the network device in the bilateral model;

[0124] Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

[0125] In conjunction with some embodiments of the second aspect, in some embodiments, the model accuracy requirement information includes at least one of the following:

[0126] The first model accuracy requirement information is used to require that the accuracy of the first AI model after training is greater than a first threshold.

[0127] The second model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than a second threshold.

[0128] The third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the third threshold.

[0129] The fourth model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the fourth threshold and greater than the fifth threshold.

[0130] The fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the sixth threshold and greater than the seventh threshold.

[0131] The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the eighth threshold.

[0132] The seventh model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the ninth threshold.

[0133] The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold.

[0134] The ninth model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the twelfth threshold and greater than the thirteenth threshold;

[0135] Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

[0136] In conjunction with some embodiments of the second aspect, in some embodiments, the first information is received via any of the following:

[0137] MAC CE;

[0138] Physical layer signaling;

[0139] RRC;

[0140] User interface;

[0141] The interface between the terminal and the network node used for AI control;

[0142] The updated applicable function report is obtained by updating the original applicable function report, which is sent to the network device by the terminal after receiving the inference configuration sent by the network device; the inference configuration is sent to the terminal by the network device after the terminal receives the terminal capability query sent by the network device and sends the terminal capability report to the network device.

[0143] In conjunction with some embodiments of the second aspect, in some embodiments, the method further includes any one of the following:

[0144] Send the model data of the first AI model to the terminal;

[0145] The model data of the first AI model sent to the OTT server;

[0146] The model data includes at least one of the training data and model parameters of the first AI model.

[0147] Thirdly, embodiments of this disclosure provide a communication device, which is a terminal, and the terminal may include at least one of a transceiver module and a processing module; wherein the terminal may be used to execute the optional implementation of the first aspect.

[0148] Fourthly, embodiments of this disclosure provide a communication device, which is a network device, and the network device may include at least one of a transceiver module and a processing module; wherein the network device may be used to execute the optional implementation of the second aspect.

[0149] Fifthly, embodiments of this disclosure provide a communication device, which is a terminal, and the terminal may include one or more processors; wherein the terminal may be used to execute an optional implementation of the first aspect.

[0150] In a sixth aspect, embodiments of this disclosure provide a communication device, which is a network device and may include one or more processors; wherein the network device may be used to perform an optional implementation of the second aspect.

[0151] In a seventh aspect, embodiments of this disclosure provide a communication system comprising: a terminal and a network device; wherein the terminal is configured to perform the method described in the optional implementation of the first aspect, and the network device is configured to perform the method described in the optional implementation of the second aspect.

[0152] Eighthly, embodiments of this disclosure provide a storage medium storing instructions that, when executed on a communication device, cause the communication device to perform the method as described in the optional implementation of the first or second aspect.

[0153] Ninthly, embodiments of this disclosure provide a program product that, when executed by a communication device, causes the communication device to perform the method as described in the optional implementation of the first or second aspect.

[0154] In a tenth aspect, embodiments of this disclosure provide a computer program that, when run on a computer, causes the computer to perform the method as described in an optional implementation of the first or second aspect.

[0155] Eleventhly, embodiments of this disclosure provide a chip or chip system. The chip or chip system includes processing circuitry configured to perform the methods described according to optional implementations of the first or second aspect above.

[0156] It is understood that the aforementioned communication equipment, terminals, network equipment, communication systems, storage media, program products, etc., are all used to execute the methods proposed in the embodiments of this disclosure. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0157] This disclosure provides a model performance verification method, a communication device, a communication system, and a storage medium. In some embodiments, the terms "model performance verification method" and "communication method," "model performance validation method," "model performance calibration method," "two-sided model performance validation method," "two-sided model performance calibration method," and "model processing method" can be used interchangeably.

[0158] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments. In all embodiments of this disclosure, unless otherwise specified or logically conflicting, the terminology and / or descriptions between the embodiments are consistent and can be mutually referenced. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0159] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0160] In this embodiment of the disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular expression or a plural expression.

[0161] In the embodiments disclosed herein, "multiple" refers to two or more.

[0162] In some embodiments, the terms “at least one of A or B, at least one of A and B”, “one or more”, “a plurality of”, “multiple”, etc., may be used interchangeably.

[0163] In some embodiments, the notation "at least one of A and B", "A and / or B", "A in one case, B in another", "in response to one case A, in response to another case B", etc., may include the following technical solutions depending on the situation: in some embodiments, A (execute A regardless of whether there is a branch B); in some embodiments, B (execute B regardless of whether there is a branch A); in some embodiments, execution is selected from A and B (A and B are selectively executed); in some embodiments, both A and B are executed. The same applies when there are more branches such as A, B, C, etc.

[0164] In some embodiments, the notation "A or B" may include the following technical solutions, depending on the situation: in some embodiments, A (execute A regardless of whether a branch B exists); in some embodiments, B (execute B regardless of whether a branch A exists); in some embodiments, execution is selected from A and B (A and B are selectively executed). The same applies when there are more branches such as A, B, and C.

[0165] The prefixes "first," "second," etc., used in the embodiments of this disclosure are merely for distinguishing different descriptive objects and do not impose restrictions on the position, order, priority, quantity, or content of the descriptive objects. The description of the descriptive objects is found in the claims or the context of the embodiments, and the use of prefixes should not constitute unnecessary restrictions. For example, if the descriptive object is a "field," the ordinal numbers preceding "field" in "first field" and "second field" do not restrict the position or order of the "fields." "First" and "second" do not restrict whether the "fields" they modify are in the same message, nor do they restrict the order of "first field" and "second field." Similarly, if the descriptive object is a "level," the ordinal numbers preceding "level" in "first level" and "second level" do not restrict the priority between "levels." Furthermore, the number of descriptive objects is not limited by ordinal numbers and can be one or more. For example, in "first device," the number of "devices" can be one or more. Furthermore, the objects modified by different prefixes can be the same or different. For example, if the object being described is "device", then "first device" and "second device" can be the same device or different devices, and their types can be the same or different. Similarly, if the object being described is "information", then "first information" and "second information" can be the same information or different information, and their content can be the same or different.

[0166] In some embodiments, “including A,” “containing A,” “for indicating A,” and “carrying A” can be interpreted as directly carrying A or indirectly indicating A.

[0167] In some embodiments, terms such as "time / frequency" and "time-frequency domain" refer to the time domain and / or frequency domain.

[0168] In some embodiments, terms such as “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “when…”, “if…”, etc. can be used interchangeably. These descriptions all refer to the device making a corresponding action under certain objective circumstances. They do not necessarily limit the time, nor do they require the device to make a judgment action when implementing it, nor do they mean that there must be other limitations.

[0169] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”.

[0170] In some embodiments, devices, etc., may be interpreted as physical or virtual, and their names are not limited to those described in the embodiments. Terms such as “device,” “equipment,” “circuit,” “network element,” “network function,” “network device,” “function,” “node,” “unit,” “section,” “system,” “network,” “chip,” “chip system,” “entity,” and “subject” are interchangeable.

[0171] In some embodiments, "network" can be interpreted as devices included in a network (e.g., access network devices, core network devices, etc.).

[0172] In some embodiments, the terms "access network device (AN device)," "radio access network device (RAN device)," "base station (BS)," "radio base station," "fixed station," "node," "access point," "transmission point (TP)," "reception point (RP)," "transmission / reception point (TRP)," "panel," "antenna panel," "antenna array," "cell," "macro cell," "small cell," "femto cell," "pico cell," "sector," "cell group," "serving cell," "carrier," "component carrier," and "bandwidth part (BWP)" can be used interchangeably.

[0173] In some embodiments, the terms "terminal", "terminal device", "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", "subscriber station", "mobile unit", "subscriber unit", "wireless unit", "remote unit", "mobile device", "wireless device", "wireless communication device", "remote device", "mobile subscriber station", "access terminal", "mobile terminal", "wireless terminal", "remote terminal", "handset", "user agent", "mobile client", and "client" can be used interchangeably.

[0174] In some embodiments, access network devices, core network devices, or network devices can be replaced by terminals. For example, embodiments of this disclosure can also be applied to structures where communication between access network devices, core network devices, or network devices and terminals is replaced by communication between multiple terminals (e.g., device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, the structure can also be configured such that the terminal has all or part of the functions of the access network device. Furthermore, terms such as "uplink" and "downlink" can be replaced with terms corresponding to communication between terminals (e.g., "sidelink"). For example, uplink channel, downlink channel, etc., can be replaced with sidelink channel, and uplink link, downlink, etc., can be replaced with sidelink link.

[0175] In some embodiments, the terminal may be replaced by an access network device, a core network device, or a network device. In this case, the access network device, core network device, or network device may also be configured to have all or some of the functions of the terminal.

[0176] In some embodiments, the acquisition of data, information, etc., may comply with the laws and regulations of the country where the location is situated.

[0177] In some embodiments, data, information, etc., may be obtained with the user's consent.

[0178] Furthermore, each element, each row, or each column in the table of this disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0179] Figure 1 is a schematic diagram of the architecture of a communication system according to an embodiment of the present disclosure. As shown in Figure 1, the communication system 100 may include a terminal 101, a network device 102, and an OTT server 103 serving the terminal 101.

[0180] In some embodiments, terminal 101 includes, for example, at least one of the following: mobile phone, wearable device, Internet of Things device, car with communication function, smart car, tablet computer, computer with wireless transceiver function, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal device in industrial control, wireless terminal device in self-driving, wireless terminal device in remote medical surgery, wireless terminal device in smart grid, wireless terminal device in transportation safety, wireless terminal device in smart city, and wireless terminal device in smart home, but is not limited thereto.

[0181] In some embodiments, the access network device 102 may be a node or device that connects a terminal to a wireless network. The access network device may include at least one of the following in a 5G communication system: an evolved Node B (eNB), a next-generation eNB (ng-eNB), a next-generation Node B (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an open RAN, a cloud RAN, a base station in other communication systems, and an access node in a Wi-Fi system, but is not limited thereto.

[0182] In some embodiments, the technical solutions of this disclosure can be applied to the Open RAN architecture. In this case, the interfaces between or within access network devices involved in the embodiments of this disclosure can be transformed into internal interfaces of Open RAN. The processes and information interactions between these internal interfaces can be implemented by software or programs.

[0183] In some embodiments, the access network device may be composed of a central unit (CU) and a distributed unit (DU). The CU may also be called a control unit. The CU-DU structure can separate the protocol layer of the access network device. Some of the protocol layer functions are centrally controlled by the CU, while the remaining part or all of the protocol layer functions are distributed in the DU and centrally controlled by the CU. However, this is not the only possibility.

[0184] In some embodiments, the core network equipment may be a single device, including a first network element, a second network element, etc., or it may be multiple devices or a group of devices, each including all or part of the first network element, the second network element, etc. Network elements may be virtual or physical. The core network may include, for example, at least one of the Evolved Packet Core (EPC), 5G Core Network (5GCN), and Next Generation Core (NGC).

[0185] It is understood that the communication system described in this disclosure is for the purpose of more clearly illustrating the technical solutions of this disclosure, and does not constitute a limitation on the technical solutions proposed in this disclosure. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions proposed in this disclosure are also applicable to similar technical problems.

[0186] The following embodiments of this disclosure can be applied to the communication system 100 shown in FIG1, or to some of the main bodies, but are not limited thereto. The main bodies shown in FIG1 are illustrative. The communication system may include all or some of the main bodies in FIG1, or may include other main bodies outside of FIG1. ​​The number and form of each main body are arbitrary. Each main body may be physical or virtual. The connection relationship between the main bodies is illustrative. The main bodies may not be connected or may be connected. The connection can be in any way, it can be a direct connection or an indirect connection, it can be a wired connection or a wireless connection.

[0187] The embodiments disclosed herein can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G new radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New radio access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), and IEEE 802.20, Ultra-Wideband (UWB), Bluetooth (a registered trademark), Public Land Mobile Network (PLMN) networks, Device-to-Device (D2D) systems, Machine-to-Machine (M2M) systems, Internet of Things (IoT) systems, Vehicle-to-Everything (V2X) systems, systems utilizing other communication methods, and next-generation systems built upon them, etc. Furthermore, multiple systems can be combined (e.g., a combination of LTE or LTE-A with 5G).

[0188] In some embodiments of this disclosure, machine learning algorithms are one of the most important implementation methods of artificial intelligence technology. Machine learning can obtain models through a large amount of training data, and these models can predict events. In many fields, models trained by machine learning can achieve very accurate prediction results.

[0189] In some embodiments, AI can be used in 5G systems for prediction and reasoning to improve system performance.

[0190] In some embodiments, when the AI ​​model is trained on the network side and the AI ​​inference is run on the terminal side, the network device needs to transmit the AI ​​model to the terminal.

[0191] In some embodiments, AI datasets can be transferred between the terminal and the network device for training.

[0192] In some embodiments, for service scenarios that use a two-sided model for CSI compression, the network device can provide the terminal with a dataset containing N1 samples, each sample including {target CSI, CSI feedback}, corresponding to the input and output of the nominal encoder, respectively.

[0193] In some embodiments, the number of N1 can reach 600,000, and each sample can have 3,000 bits. Therefore, the data size of a dataset can be 600K*(2000bits+1000bits) / (8bits / Byte)=225MB.

[0194] In some embodiments, during the data transmission process, the terminal collects the data set for the terminal's one-sided model according to the network configuration and transmits it to the OTT server serving the terminal through the 3GPP network.

[0195] In some embodiments, for a two-sided model, after sending the dataset and / or model parameters to the terminal or the OTT server on the terminal side, the terminal's OTT server needs to retrain the model to pair with the network-side model for CSI compression.

[0196] In some embodiments, bilateral model performance verification and bilateral model performance calibration are required before the model retrained on the OTT server on the terminal side can be used for inference on the terminal side.

[0197] In view of this, the present disclosure proposes a model performance verification method, communication device, communication system and storage medium to realize the performance verification of the first AI model by the terminal and network device, thereby ensuring that the AI ​​model applied to the terminal can be used normally and improving the reliability of the AI ​​model in the two-sided model business scenario.

[0198] Figure 2A is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure. This method can be executed by the aforementioned communication system. As shown in Figure 2A, the method may include:

[0199] Step S2101: Network device 102 sends second information to terminal 101.

[0200] In some embodiments, terminal 101 receives second information. For example, terminal 101 receives second information sent by network device 102.

[0201] In some embodiments, the second information is used to query the capabilities of terminal 101.

[0202] In some embodiments, network device 102 can dynamically adapt to the capabilities of terminal 101 by querying the capabilities of terminal 101.

[0203] For example, network device 102 sends second information to terminal 101 to determine whether terminal 101 has the ability to run the CSI compression model.

[0204] For example, network device 102 sends second information to terminal 101 to determine the current real-time capabilities of terminal 101 (e.g., remaining battery power, CPU load, etc.). Based on the current real-time capabilities of terminal 101, network device 102 can dynamically adjust the tasks assigned to terminal 101, thereby achieving resource optimization.

[0205] In some embodiments, the name of the second information is not limited, such as UE Capability Enquiry Request, Terminal Capability Request, Capability Configuration Query Request, etc.

[0206] In some embodiments, when network device 102 and terminal 101 need to collaboratively perform services based on an AI model, network device 102 needs to send second information to terminal 101 to obtain the types of AI models supported by the terminal and determine whether the AI ​​models supported by the terminal are suitable for the current business scenario.

[0207] For example, in a CSI compression service scenario, terminal 101 and network device 102 need to collaborate to compress and restore CSI. That is, terminal 101 needs to perform CSI compression using the CSI compression model, and network device 102 needs to restore the CSI compression result. Therefore, before performing the CSI compression service, network device 102 needs to send a second message to terminal 101 to query whether terminal 101 supports CSI compression.

[0208] In some embodiments, when terminal 101 first accesses network device 102, network device 102 does not store the capability information of terminal 101. At this time, network device 102 sends second information to terminal 101 to actively query the capabilities of terminal 101.

[0209] In some embodiments, when the capabilities of terminal 101 change, such as when a hardware or software upgrade of terminal 101 results in an update of its capabilities, network device 102 needs to send second information to terminal 101 to reacquire the latest capability information of terminal 101.

[0210] In some embodiments, when the network device 102 is configured or its functions are upgraded, it is necessary to verify whether the terminal 101 is compatible. At this time, the network device 102 needs to send second information to the terminal 101 to verify whether the terminal 101 has the corresponding capabilities.

[0211] In some embodiments, when terminal 101 switches to a new cell, the network device 102 of the new cell needs to send second information to terminal 101 to reacquire the capabilities of terminal 101 in order to optimize resource configuration.

[0212] Step S2102: Terminal 101 sends third information to network device 102.

[0213] In some embodiments, after receiving the second information, the terminal 101 sends the third information to the network device 102.

[0214] In some embodiments, network device 102 receives third information. For example, network device 102 receives third information sent by terminal 101.

[0215] In some embodiments, the third information carries capability information of the terminal 101.

[0216] In some embodiments, network device 102 can achieve accurate network adaptation and resource optimization based on the capability information of terminal 101.

[0217] For example, network device 102 determines that terminal 101 has the capability to support CSI compression model based on the capability information of terminal 101, and then configures the corresponding CSI compression model parameters for terminal 101, thereby working with terminal 101 to achieve CSI compression.

[0218] For example, network device 102 optimizes spectrum utilization to improve data transmission efficiency based on the wireless access capabilities of terminal 101 (such as the number of MIMO antennas and carrier aggregation support).

[0219] In some embodiments, the name of the second information is not limited, and may be, for example, UE Capability Reporting, terminal capability information, terminal capability configuration, etc.

[0220] In some embodiments, terminal 101 receives second information sent by network device 102 and sends third information to network device 102.

[0221] In some embodiments, terminal 101 receives second information sent by network device 102, and terminal 101 has the capability required in the second information to send third information to network device 102.

[0222] In some embodiments, terminal 101 may actively send third information to network device 102, in which case step S2101 can be omitted.

[0223] For example, when the terminal 101 switches to a new cell, it can proactively send third information to the network device 102.

[0224] For example, after a hardware or software upgrade, terminal 101 proactively sends third information to network device 102.

[0225] Step S2103: Network device 102 sends fourth information to terminal 101.

[0226] In some embodiments, after receiving the third information, the network device sends the fourth information to the terminal 101.

[0227] In some embodiments, terminal 101 receives fourth information. For example, terminal 101 receives fourth information sent by network device 102.

[0228] In some embodiments, the fourth information is information associated with the model reasoning task.

[0229] In some embodiments, the fourth information is used to indicate the deployment parameters and / or execution strategy of the AI ​​model inference task.

[0230] In some embodiments, the fourth information is used to indicate the hardware and / or software parameters for the deployment of the AI ​​model.

[0231] For example, the fourth piece of information includes the loading path of the AI ​​model, the version information of the AI ​​model, and the hardware resource allocation strategy.

[0232] In some embodiments, the name of the second information is not limited, and may be, for example, inference configuration, inference parameters, etc.

[0233] In some embodiments, terminal 101 obtains the fourth information specified by the protocol, in which case step S2103 can be omitted.

[0234] In some embodiments, the terminal 101 obtains the fourth information from the upper layer(s), in which case step S2103 can be omitted.

[0235] In some embodiments, the terminal 101 processes the information to obtain the fourth information, in which case step S2103 can be omitted.

[0236] In some embodiments, the terminal 101 autonomously implements the function indicated by the fourth information, or the above function is a default or default value, in which case step S2103 can be omitted.

[0237] Step S2104: Terminal 101 sends the fifth information to network device 102.

[0238] In some embodiments, after receiving the fourth information, the terminal 101 sends the fifth information to the network device 102.

[0239] In some embodiments, network device 102 receives fifth information. For example, network device 102 receives fifth information sent by terminal 101.

[0240] In some embodiments, the fifth information is used to enable resource adaptation and policy optimization in collaboration between terminal 101 and network device 102.

[0241] For example, the fifth information includes at least one of the following: the hardware capabilities supported by terminal 101, protocol compatibility, and the mechanism of service processing strategy.

[0242] In some embodiments, the name of the fifth piece of information is not limited, and may be, for example, applicable functionality reporting, applicable functionality information, applicable functionality configuration, etc.

[0243] In some embodiments, terminal 101 sends fifth information to network device 102, but terminal 101 is in an applicable but available state.

[0244] In some embodiments, an applicable but unavailable state means that terminal 101 supports a certain capability (or function), but the capability (or function) is unavailable due to limitations of the current environment or resource conditions.

[0245] For example, in a scenario where CSI compression is performed collaboratively using a bilateral model, terminal 101's capabilities support the CSI compression model. However, because terminal 101 lacks the relevant model parameters for the CSI compression model, it cannot perform CSI compression using the CSI compression model. In this case, terminal 101 sends a fifth message to network device 102 to indicate that terminal 101 should apply the CSI compression model, but terminal 101 is in an applicable but unavailable state.

[0246] Step S2105: Network device 102 sends the sixth information to terminal 101.

[0247] In some embodiments, after receiving the fifth information, the network device 102 sends the sixth information to the terminal 101.

[0248] In some embodiments, terminal 101 receives sixth information. For example, terminal 101 receives sixth information sent by network device 102.

[0249] In some embodiments, the sixth information is model data of the AI ​​model. For example, network device 102 sends the model data of the AI ​​model to terminal 101 according to the actual business scenario; wherein, the model data of the AI ​​model may be the model data of the first AI model applied to terminal 101 in a bilateral model, or it may be the model data of the second AI model applied to network device 102.

[0250] In some embodiments, model data includes at least one of a dataset used to train the AI ​​model and model parameters.

[0251] For example, in a CSI compression service scenario, network device 102 sends sixth information to terminal 101; wherein, the sixth information includes the dataset and model parameters corresponding to the CSI compression model applied to terminal 101. The dataset, for example, consists of N1 samples, each sample including {target CSI, CSI feedback}.

[0252] In some embodiments, {target CSI, CSI feedback} is also referred to as the first CSI and the first CSI feedback; in subsequent embodiments, {target CSI, CSI feedback} can also be understood as the first CSI and the first CSI feedback. To avoid repetition, this will not be repeated hereafter.

[0253] In some embodiments, the name of the sixth information is not limited, such as model data, model parameter information, model configuration information, etc.

[0254] Step S2106: Network device 102 sends the seventh information to terminal 101.

[0255] In some embodiments, terminal 101 receives seventh information. For example, terminal 101 receives seventh information sent by network device 102.

[0256] In some embodiments, the seventh information is used by the terminal 101 to evaluate the model performance of the AI ​​model.

[0257] In some embodiments, the seventh information is used by the terminal 101 to evaluate the model performance of the model applied to the terminal 101 in the bilateral model (also known as the first AI model).

[0258] In some embodiments, the seventh information is used to evaluate whether the model performance of the first AI model meets the performance requirements.

[0259] In some embodiments, performance requirements are used to indicate the conditions required for the model performance of the first AI model to achieve the desired performance.

[0260] In some embodiments, the first AI model is used to compress the CSI.

[0261] In some embodiments, the seventh information and the sixth information in step S2105 need to be sent to terminal 101 independently. For example, the seventh information and the sixth information can be carried in different signaling messages.

[0262] In some embodiments, the network device sends the seventh information to the terminal 101 via the Uu interface.

[0263] For example, network device 102 (e.g., base station or network side) first collects the seventh information, and then sends the collected seventh information to terminal 101 through Uu interface.

[0264] For example, network device 102 (e.g., base station or network side) collects seventh information from the server or node training the model, and then sends the collected seventh information to terminal 101 through the Uu interface.

[0265] In some embodiments, the name of the seventh information is not limited, such as accuracy target, performance requirement information, etc.

[0266] In some embodiments, the performance requirement information is used to evaluate whether the model performance of the first AI model meets the performance requirements.

[0267] In some embodiments, the seventh information includes model validation requirements and model accuracy requirements.

[0268] In some embodiments, model validation requirements include information for validating the performance of a first AI model.

[0269] In some embodiments, model accuracy requirement information is used to indicate the accuracy requirement conditions under which the model performance achieves the desired performance.

[0270] In some embodiments, model validation requirements include at least one of the following:

[0271] 1), {target CSI, CSI feedback};

[0272] 2) At least one of the following: target CSI and terminal 101 side model (also known as the first AI model, which can also be understood as the first AI model in subsequent embodiments; to avoid repetition, it will not be described again later), model parameters, and model structure trained by network device 102 (e.g., base station).

[0273] 3) At least one of the following: target CSI and network device 102 (e.g., base station) model, model parameters, and model structure.

[0274] In some embodiments, model validation requirements include at least one of the following:

[0275] a) First CSI and First CSI Feedback;

[0276] b) First CSI and first model information; the first model information includes at least one of the following: the first AI model trained by the network device 102, the model parameters of the first AI model trained by the network device 102, and the model structure of the first AI model trained by the network device 102.

[0277] c) First CSI and second model information; the second model information includes at least one of the following: second AI model, model parameters of the second AI model, and model structure of the second AI model; the second AI model is the AI ​​model applied to the network device 102 in the bilateral model;

[0278] Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

[0279] In some embodiments, the model accuracy requirement information includes at least one of the following:

[0280] 1) The accuracy of the training model on the terminal 101 side (accuracy can be expressed as a percentage, such as 80% accuracy) is higher than a certain threshold (a certain threshold is, for example, 60%).

[0281] 2) The difference between the CSI output by the training model on the terminal 101 side and the target CSI after pairing with the received network side model is less than a certain threshold;

[0282] The network-side model is also known as the second AI model, and in subsequent embodiments, the network-side model can also be referred to as the second AI model.

[0283] The CSI output after the training model on the terminal 101 side is paired with the received network-side model is also called the second CSI. In subsequent embodiments, it can also be referred to as the second CSI. To avoid repetition, it will not be described again.

[0284] 3) The average value of the CSI output by the training model on the terminal 101 side and the target CSI after being paired with the received network side model is less than a certain threshold;

[0285] 4) The difference between the CSI output by the training model on the terminal 101 side and the target CSI after pairing with the received network side model is less than a certain threshold and greater than a certain threshold.

[0286] 5) The average value of the CSI and target CSI output by the training model on the terminal 101 side after being paired with the received network side model is less than a certain threshold and greater than a certain threshold.

[0287] 6) The difference between the CSI feedback obtained from training the model on the terminal 101 side and the CSI feedback in the model validation requirement information is less than a certain threshold;

[0288] The CSI feedback obtained by training the model on the terminal 101 side is also called the second CSI feedback. In the following embodiments, it can also be called the second CSI feedback. To avoid repetition, it will not be described again.

[0289] 7) The average value of the CSI feedback obtained from training the model on the terminal 101 side and the CSI feedback in the model validation requirement information is less than a certain threshold;

[0290] 8) The difference between the CSI feedback obtained from training the model on the terminal 101 side and the CSI feedback in the model validation requirement information is less than a certain threshold and greater than a certain threshold.

[0291] 9) The average value of the CSI feedback obtained from training the model on the terminal 101 side and the CSI feedback in the model validation requirement information is less than a certain threshold and greater than a certain threshold.

[0292] In some embodiments, the model accuracy requirement information includes at least one of the following:

[0293] a) First model accuracy requirement information, used to require that the accuracy of the first AI model after training is greater than a first threshold;

[0294] b) Second model accuracy requirement information, which requires that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal 101 is less than a second threshold.

[0295] c) Third model accuracy requirement information, which is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal 101 is less than the third threshold;

[0296] d) Fourth model accuracy requirement information, which requires that the difference between the second CSI output by the first AI model after training and the second AI model received by the terminal 101 and the first CSI is less than the fourth threshold and greater than the fifth threshold.

[0297] e) Fifth model accuracy requirement information, which requires that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal 101 is less than the sixth threshold and greater than the seventh threshold;

[0298] f) The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the eighth threshold.

[0299] g) The seventh model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the ninth threshold.

[0300] h) The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold.

[0301] i) Ninth model accuracy requirement information, which requires that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the twelfth threshold and greater than the thirteenth threshold;

[0302] Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

[0303] In some embodiments, the model validation requirement information is used by terminal 101 to validate the performance of the first AI model.

[0304] For example, terminal 101 can verify whether the model parameters of the first AI model are aligned between terminal 101 and network device 102, whether the functions of the first AI model are complete, and whether terminal 101 and network device 102 are compatible with the first AI model based on model verification requirement information, so as to ensure the consistency and compatibility of the first AI model between terminal 101 and network device 102, thereby ensuring the availability of the first AI model.

[0305] In some embodiments, the model accuracy requirement information is used by terminal 101 to calibrate the accuracy of the first AI model.

[0306] For example, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information to ensure that the accuracy of the first AI model meets the requirements and improves the accuracy of the first AI model.

[0307] In some embodiments, terminal 101 verifies the performance of the first AI model to ensure its consistency, functional integrity, and compatibility; it calibrates the accuracy of the first AI model to optimize it, thereby ensuring its performance and accuracy; through the above methods, efficient collaboration between terminal 101 and network device 102 on the first AI model is achieved.

[0308] In some embodiments, the seventh information is carried in any of the following:

[0309] MAC CE;

[0310] Physical layer signaling;

[0311] RRC;

[0312] User interface;

[0313] Interface between the terminal and the network node used for AI control.

[0314] In some embodiments, the above steps S2101-S2106 can be referred to as the model transfer process stage.

[0315] Step S2107: Terminal 101 sends the sixth message to OTT server 103.

[0316] In some embodiments, the sixth information includes model data applied to the first AI model of the terminal 101, including at least one of the dataset used to train the first AI model and the model parameters of the first AI model.

[0317] In some embodiments, OTT server 103 is an OTT server serving terminal 101.

[0318] In some embodiments, the OTT server 103 is used to train a first AI model.

[0319] Step S2108: OTT server 103 trains the first AI model.

[0320] In some embodiments, the OTT server 103 trains the first AI model based on the model data of the first AI model in the sixth information.

[0321] Step S2109: Terminal 101 downloads the trained first AI model.

[0322] In some embodiments, in steps S2108 and S2109, the OTT server 103 trains the first AI model, and the terminal 101 downloads the trained first AI model, also known as: Model retraining at UE-side OTT server and download.

[0323] In step S2110, terminal 101 evaluates the performance of the first AI model.

[0324] In some embodiments, terminal 101 evaluates the model performance of the first AI model based on the seventh information.

[0325] In some embodiments, the terminal 101 evaluates the model performance of the first AI model based on performance requirement information.

[0326] In some embodiments, the terminal 101 evaluates the model performance of the AI ​​model to determine whether the model performance of the first AI model has reached the performance target.

[0327] In some embodiments, the performance requirement information is used to evaluate whether the model performance of the first AI model meets the performance requirements.

[0328] In some embodiments, the first AI model reaching the performance target can be understood as the model performance of the first AI model meeting the performance requirements. That is, when the model performance of the first AI model meets the performance requirements, the terminal 101 sends the first information to the network device 102.

[0329] In some embodiments, performance requirements are used to indicate the conditions required for the model performance to achieve the desired performance.

[0330] In some embodiments, before sending the first information to the network device 102, the terminal 101 also needs to evaluate the model performance of the first AI model. If the model performance of the first AI model meets the performance requirements, the terminal 101 sends the first information to the network device 102.

[0331] In some embodiments, the terminal 101 needs to obtain the performance requirement information of the first AI model, and then evaluate the model performance of the first AI model based on the performance requirement information.

[0332] In some embodiments, terminal 101 receives seventh information sent by network device 102 and evaluates the model performance of the first AI model based on the seventh information.

[0333] In some embodiments, terminal 101 obtains seventh information from OTT server 103.

[0334] In some embodiments, terminal 101 receives seventh information from network device sent by OTT server 103.

[0335] In some embodiments, terminal 101 receives seventh information sent by OTT server 103, wherein the seventh information is sent to terminal 101 by OTT after receiving the seventh information sent by network device.

[0336] For example, network device 102 sends the seventh information to OTT server 103, and OTT server 103 sends the seventh information to terminal 101.

[0337] In some embodiments, the name of the seventh information is not limited, such as accuracy target, performance requirement information, etc.

[0338] In some embodiments, the terminal 101 evaluates the model performance of the first AI model based on performance requirement information.

[0339] In some embodiments, the relevant content regarding the seventh information can be found in the embodiment corresponding to step S106 in Figure 2A.

[0340] In some embodiments, the terminal 101 evaluates the model performance of the first AI model based on performance requirement information, including at least one of the following:

[0341] a) Terminal 101 verifies the performance of the first AI model based on the model verification requirements information.

[0342] b) Terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information.

[0343] In some embodiments, successful performance verification of the first AI model and / or meeting the accuracy requirements of the first AI model indicate that the first AI model has achieved the performance target (also known as meeting the performance requirements).

[0344] In some embodiments, the terminal 101 verifies the performance of the first AI model based on model verification requirement information, which can be achieved in the following ways:

[0345] Terminal 101 verifies the performance of the first AI model based on the first CSI and the first CSI feedback.

[0346] For example, in a CSI compression scenario, terminal 101 inputs the first CSI into the trained first AI model to verify whether the trained first AI model can obtain the first CSI feedback. If the first CSI feedback can be obtained, the performance verification of the first AI model is successful; if the first CSI feedback cannot be obtained, the performance verification of the first AI model fails.

[0347] In some embodiments, the terminal 101 verifies the performance of the first AI model based on model verification requirement information, which can be achieved in the following ways:

[0348] Terminal 101 verifies the performance of the first AI model based on the first CSI and the first model information; wherein, the first model information includes at least one of the following: the first AI model trained by network device 102, the model parameters of the first AI model trained by network device 102, and the model structure of the first AI model trained by network device 102.

[0349] For example, in a CSI compression service scenario, terminal 101 obtains a first AI model trained by network device 102 based on the first model information. Then, terminal 101 inputs the first CSI into the first AI model trained by OTT server 103 to obtain a first CSI compression result; it then inputs the first CSI into the first AI model trained by network device 102 to obtain a second CSI compression result. The first CSI compression result and the second CSI compression result are compared to see if they are the same; if they are the same, the performance verification of the first AI model is successful; if they are different, the performance verification of the first AI model fails.

[0350] In some embodiments, the terminal 101 verifies the performance of the first AI model based on model verification requirement information, which can be achieved in the following ways:

[0351] Terminal 101 verifies the performance of the first AI model based on the first CSI and the second model information; the second model information includes at least one of the following: the second AI model, the model parameters of the second AI model, and the model structure of the second AI model; the second AI model is the AI ​​model applied to network device 102 in the bilateral model.

[0352] For example, in a CSI compression service scenario, terminal 101 obtains the second AI model of network device 102 based on the second model information. Terminal 101 inputs the first CSI into the first AI model for CSI compression, obtaining the CSI compression result output by the first AI model; then, it inputs the CSI compression result into the second AI model to verify whether the second AI model can reconstruct the first CSI. If the second AI model can reconstruct the first CSI, the performance verification of the first AI model is successful; if the second AI model cannot reconstruct the first CSI, the performance verification of the first AI model fails.

[0353] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0354] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirement information of the first model.

[0355] In some embodiments, the first model accuracy requirement information is used to require that the accuracy of the trained first AI model be greater than a first threshold.

[0356] For example, the N1 samples are divided into a training dataset and a validation dataset. The OTT server 103 trains the first AI model using the training dataset, and the terminal 101 determines the accuracy of the trained first AI model based on the validation dataset. If the accuracy of the trained first AI model is greater than a first threshold, the accuracy of the trained first AI model meets the standard; if the accuracy of the trained first AI model is less than the first threshold, the accuracy of the trained first AI model does not meet the standard.

[0357] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0358] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirements of the second model.

[0359] In some embodiments, the second model accuracy requirement information is used to require that the difference between the second CSI output by the first AI model after training and the second AI model received by the terminal 101 and the first CSI is less than a second threshold.

[0360] For example, terminal 101 pairs the first AI model trained by OTT server 103 with the second AI model of network device 102. Then, it calculates the difference between the second CSI output by the first AI model and the first CSI in the model accuracy requirement information. If the difference between the second CSI and the first CSI is less than a second threshold, the accuracy of the trained first AI model meets the requirement; if the difference between the second CSI and the first CSI is greater than the second threshold, the accuracy of the trained first AI model does not meet the requirement.

[0361] In some embodiments, the second CSI can be understood as the CSI compression result output by the first AI model.

[0362] In some embodiments, the second CSI is obtained by the terminal 101 acquiring the CSI in the channel in real time and inputting it into the first AI model, and the first AI model compressing the real-time CSI.

[0363] In some embodiments, the second CSI is obtained by the terminal 101 inputting the first CSI into the first AI model, and the first AI model compressing the first CSI.

[0364] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0365] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirements of the third model.

[0366] In some embodiments, the third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and pairing with the second AI model received by the terminal 101 is less than a third threshold.

[0367] For example, terminal 101 pairs the first AI model trained by OTT server 103 with the second AI model of network device 102. Then, it calculates the average value between the second CSI output by the first AI model and the first CSI in the model accuracy requirement information. If the average value of the second CSI and the first CSI is less than a third threshold, the accuracy of the trained first AI model meets the requirement; if the average value of the second CSI and the first CSI is greater than the third threshold, the accuracy of the trained first AI model does not meet the requirement.

[0368] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0369] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirements of the fourth model.

[0370] In some embodiments, the fourth model accuracy requirement information is used to require that the difference between the second CSI output by the first AI model after training and the second AI model received by the terminal 101 is less than a fourth threshold and greater than a fifth threshold.

[0371] For example, terminal 101 pairs the first AI model trained by OTT server 103 with the second AI model of network device 102. Then, it calculates the difference between the second CSI output by the first AI model and the first CSI in the model accuracy requirement information. If the difference between the second CSI and the first CSI is less than a fourth threshold and greater than a fifth threshold, the accuracy of the trained first AI model meets the standard; otherwise, the accuracy of the trained first AI model does not meet the standard.

[0372] In some embodiments, the fourth threshold can be represented by a specific numerical value, for example, the fourth threshold is 5.

[0373] In some embodiments, the fourth threshold can be represented as a percentage of a certain value or a percentage of a certain threshold, for example, the fourth threshold is 5*20%.

[0374] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0375] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirements of the fifth model.

[0376] In some embodiments, the fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output after the first AI model after training is paired with the second AI model received by the terminal 101 is less than a sixth threshold and greater than a seventh threshold.

[0377] For example, terminal 101 pairs the first AI model trained by OTT server 103 with the second AI model of network device 102. Then, it calculates the average value between the second CSI output by the first AI model and the first CSI in the model accuracy requirement information. If the average value of the second CSI and the first CSI is less than a sixth threshold and the difference is greater than a seventh threshold, then the accuracy of the trained first AI model meets the standard; otherwise, the accuracy of the trained first AI model does not meet the standard.

[0378] In some embodiments, the fifth threshold can be represented by a specific numerical value, for example, the fifth threshold is 10.

[0379] In some embodiments, the fifth threshold can be represented as a percentage of a certain value or a percentage of a certain threshold, for example, the fifth threshold is 10*30%.

[0380] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0381] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirements of the sixth model.

[0382] In some embodiments, the sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than an eighth threshold.

[0383] For example, terminal 101 acquires the original CSI in the channel, inputs the original CSI into the trained first AI model for CSI compression, and obtains the second CSI feedback based on the CSI compression result. Terminal 101 compares the second CSI feedback with the first CSI feedback in the model accuracy requirement information. If the difference between the second CSI feedback and the first CSI feedback is less than an eighth threshold, the accuracy of the trained first AI model meets the standard; otherwise, the accuracy of the trained first AI model does not meet the standard.

[0384] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0385] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirements of the seventh model.

[0386] In some embodiments, the seventh model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the trained first AI model be less than a ninth threshold.

[0387] For example, terminal 101 acquires the original CSI in the channel, inputs the original CSI into the trained first AI model for CSI compression, and obtains the second CSI feedback based on the CSI compression result. Terminal 101 compares the second CSI feedback with the first CSI feedback in the model accuracy requirement information. If the average of the second CSI feedback and the first CSI feedback is less than the ninth threshold, the accuracy of the trained first AI model meets the standard; otherwise, the accuracy of the trained first AI model does not meet the standard.

[0388] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0389] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirement information of the eighth model.

[0390] In some embodiments, the eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the trained first AI model is less than the tenth threshold and greater than the eleventh threshold.

[0391] For example, terminal 101 acquires the original CSI from the channel, inputs the original CSI into the trained first AI model for CSI compression, and obtains second CSI feedback based on the CSI compression result. Terminal 101 compares the second CSI feedback with the first CSI feedback in the model accuracy requirement information. If the difference between the second CSI feedback and the first CSI feedback is less than the tenth threshold and greater than the eleventh threshold, then the accuracy of the trained first AI model meets the standard; otherwise, the accuracy of the trained first AI model does not meet the standard.

[0392] In some embodiments, the tenth threshold can be represented by a specific numerical value, for example, the tenth threshold is 20.

[0393] In some embodiments, the tenth threshold can be represented as a percentage of a certain value or a percentage of a certain threshold, for example, the tenth threshold is 20*50%.

[0394] In some embodiments, the terminal 101 calibrates the accuracy of the first AI model based on the model accuracy requirement information, which can be achieved in the following ways:

[0395] Terminal 101 calibrates the accuracy of the first AI model based on the accuracy requirements of the ninth model.

[0396] In some embodiments, the ninth model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the trained first AI model be less than the twelfth threshold and greater than the thirteenth threshold.

[0397] For example, terminal 101 acquires the original CSI from the channel, inputs the original CSI into the trained first AI model for CSI compression, and obtains the second CSI feedback based on the CSI compression result. Terminal 101 compares the second CSI feedback with the first CSI feedback in the model accuracy requirement information. If the average of the second CSI feedback and the first CSI feedback is less than the twelfth threshold and greater than the thirteenth threshold, then the accuracy of the trained first AI model meets the standard; otherwise, the accuracy of the trained first AI model does not meet the standard.

[0398] In some embodiments, the twelfth threshold can be represented by a specific numerical value, for example, the twelfth threshold is 50.

[0399] In some embodiments, the twelfth threshold can be represented as a percentage of a certain value or a percentage of a certain threshold, for example, the twelfth threshold is 50*20%.

[0400] Step S2111: Terminal 101 sends first information to network device 102.

[0401] In some embodiments, the first information is used to indicate that a first executable AI model is configured on the terminal 101.

[0402] In some embodiments, an executable first AI model is configured on the terminal 101, which can be understood as the terminal 101 having an executable first AI model deployed on it, or the terminal 101 having an executable first AI model.

[0403] In some embodiments, the first information is sent to the network device via any of the following:

[0404] 1) MAC CE;

[0405] 2) Physical layer signaling;

[0406] 3) RRC;

[0407] 4) User interface;

[0408] 5) Interface between the terminal and the network node used for AI control;

[0409] 6) Update of applicable functionality reporting; The updated applicable functionality reporting is obtained by updating the applicable functionality reporting, which is sent by the terminal 101 to the network device 102 after receiving the inference configuration sent by the network device 102; The inference configuration is sent by the network device 102 to the terminal 101 after the terminal 101 receives the UE Capability Enquiry sent by the network device 102 and sends the UE Capability Report to the network device 102.

[0410] In some embodiments, the name of the first information is not limited, and may be, for example, model applicable function information, model applicable function configuration, etc.

[0411] In some embodiments, the signaling name carrying the first information is not limited. For example, the signaling name may be model availability reporting, update applicable function report, etc.

[0412] In some embodiments, terminal 101 sends first information to network device 102 via a separate RRC message (e.g., model availability reporting). That is, the first information is carried in a separate RRC message.

[0413] In some embodiments, terminal 101 sends an updated applicable function report to network device 102 by updating the applicable function report, wherein the updated applicable function report carries first information.

[0414] In some embodiments, the first information may be reported as a new separate RRC message (e.g., model availability reporting) or by updating applicable functionality reporting to report that terminal 101 has an executable model.

[0415] In some embodiments, terminal 101 sends an updated applicable function report to network device 102 only if the first AI model achieves the performance target.

[0416] Step S2112: Network device 102 sends the eighth information to terminal 101.

[0417] In some embodiments, terminal 101 receives eighth information. For example, terminal 101 receives eighth information sent by network device 102.

[0418] In some embodiments, the eighth information is used to indicate the activation of the first AI model.

[0419] In some embodiments, network device 102 sends eighth information to terminal 101 based on performance monitoring.

[0420] In some embodiments, the network device 102 sends the eighth information based on performance monitoring, which means that the network device 102 dynamically activates the first AI model according to real-time performance monitoring data, aiming to optimize terminal and network resources.

[0421] In some embodiments, the name of the eighth message is not limited, and may be, for example, an activation instruction, a model activation instruction, a model activation message, etc.

[0422] For example, network device 102 sends an activation command based on performance monitoring.

[0423] In some embodiments, the names of information, etc., are not limited to the names described in the embodiments. Terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "domain", "field", "symbol", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", and "chip" can be used interchangeably.

[0424] In some embodiments, "acquire," "get," "obtain," "receive," "transmit," "bidirectional transmission," and "send and / or receive" can be used interchangeably and can be interpreted as receiving from other entities, acquiring from protocols, acquiring from higher layers, obtaining through self-processing, or autonomous implementation. Protocols include, for example, at least one of the 3GPP protocol, Wi-Fi protocol, and audio and / or video protocols.

[0425] In some embodiments, terms such as “send,” “transmit,” “report,” “distribute,” “transfer,” “bidirectional transmission,” “send and / or receive” can be used interchangeably.

[0426] In some embodiments, terms such as "certain," "preset," "default," "set," "indicated," "a certain," "any," and "first" can be used interchangeably. "Certain A," "preset A," "default A," "set A," "indicated A," "a certain A," "any A," and "first A" can be interpreted as A pre-defined in a protocol or the like, or as A obtained through setting, configuration, or instruction, or as specific A, a certain A, any A, or first A, but are not limited thereto.

[0427] In some embodiments, if an arrow in the interaction diagram representing the sending of information, signaling, etc. from one subject to another passes through other subjects, it can be interpreted as the information being forwarded from one subject to another via other subjects, or it can be interpreted as the information being sent from one subject to another without passing through other subjects.

[0428] The model performance verification method disclosed in this embodiment may include at least one of steps S2101 to S2112. For example, step S2106 may be implemented as an independent embodiment, step S2111 may be implemented as an independent embodiment, steps S2106 and S2111 may be implemented as independent embodiments, and steps S2101, S2102, S2103, S2104, S2105, S2107, S2108, S2109, S2110, S2111, and S2112 may be implemented as independent embodiments, but are not limited thereto.

[0429] In some embodiments, the order of any two steps in steps S2101 to S2112 can be interchanged or they can be performed simultaneously.

[0430] In some embodiments, steps S2105 and S2106 may be performed in an alternate order or simultaneously.

[0431] In some embodiments, steps S2101 to S2109 and step S2112 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0432] In some embodiments, steps S2101 to S2105, S2107, S2108, S2109 and S2112 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0433] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0434] Figure 2B is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 2B, the embodiment of the present disclosure relates to a model performance verification method, which is executed by a communication system 100. The model performance verification method includes at least one of the following steps:

[0435] Step S2201: Network device 102 sends second information to terminal 101.

[0436] The optional implementation of step S2201 can be found in the optional implementation of step S2101 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0437] Step S2202: Terminal 101 sends third information to network device 102.

[0438] The optional implementation of step S2202 can be found in the optional implementation of step S2102 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0439] Step S2203: Network device 102 sends fourth information to terminal 101.

[0440] The optional implementation of step S2203 can be found in the optional implementation of step S2103 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0441] Step S2204: Terminal 101 sends the fifth information to network device 102.

[0442] The optional implementation of step S2204 can be found in the optional implementation of step S2104 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0443] Step S2205: Network device 102 sends the sixth message to OTT server 103.

[0444] In some embodiments, after receiving the fifth information, the network device 102 sends the sixth information to the OTT server 103 serving the terminal 101.

[0445] In some embodiments, the OTT server 103 receives sixth information. For example, the OTT server 103 receives sixth information sent by the network device 102.

[0446] The optional implementation of step S2205 can be found in the optional implementation of step S2105 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0447] Step S2206: Network device 102 sends the seventh message to OTT server 103.

[0448] In some embodiments, the seventh information and the sixth information in step S2205 need to be sent to the OTT server 103 independently. Exemplarily, the seventh information and the sixth information can be carried in different signaling messages.

[0449] In some embodiments, optional implementations of step S2206 can be found in optional implementations of step S2106 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0450] Step S2207: OTT server 103 sends the seventh message to terminal 101.

[0451] In some embodiments, terminal 101 may send a seventh information acquisition request to OTT server 103, and OTT server 103 shall send the seventh information to terminal 101 after receiving the seventh information acquisition request.

[0452] In some embodiments, after receiving the seventh information sent by the network device 102, the OTT server 103 sends the seventh information to the terminal 101.

[0453] In some embodiments, the OTT server 103 may periodically send the seventh information to the terminal 101.

[0454] In some embodiments, terminal 101 may also directly receive the seventh information sent by network device 102.

[0455] In some embodiments, optional implementations of step S2207 can be found in optional implementations of step S2107 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0456] Step S2208: Train the first AI model on OTT server 103.

[0457] The optional implementation of step S2208 can be found in the optional implementation of step S2108 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0458] Step S2209: Terminal 101 downloads the first trained AI model.

[0459] In some embodiments, steps S2207 and S2209 may be performed in an alternate order or simultaneously.

[0460] In step S2210, terminal 101 evaluates the performance of the first AI model.

[0461] In some embodiments, optional implementations of step S2210 can be found in optional implementations of step S2110 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0462] Step S2211: Terminal 101 sends first information to network device 102.

[0463] In some embodiments, optional implementations of step S2211 can be found in the optional implementations of step S2111 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0464] Step S2212: Network device 102 sends the eighth information to terminal 101.

[0465] In some embodiments, optional implementations of step S2212 can be found in optional implementations of step S2112 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0466] The model performance verification method disclosed in this embodiment may include at least one of steps S2201 to S2212. For example, step S2206 may be implemented as an independent embodiment, step S2211 may be implemented as an independent embodiment, steps S2206 and S2211 may be implemented as independent embodiments, and steps S2201, S2202, S2203, S2204, S2205, S2207, S2208, S2209, S2210, S2211 and S2212 may be implemented as independent embodiments, but are not limited thereto.

[0467] In some embodiments, the order of any two steps in steps S2201 to S2212 can be interchanged or they can be performed simultaneously.

[0468] In some embodiments, steps S2205 and S2206 may be performed in an alternate order or simultaneously.

[0469] In some embodiments, steps S2201 to S2209 and step S2212 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0470] In some embodiments, steps S2201 to S2205, S2207, S2208, S2209 and S2212 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0471] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0472] Figure 2C is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 2C, the embodiment of the present disclosure relates to a model performance verification method, which is executed by a communication system 100. The model performance verification method includes at least one of the following steps:

[0473] Step S2301: Network device 102 sends second information to terminal 101.

[0474] In some embodiments, optional implementations of step S2301 can be found in optional implementations of step S2101 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0475] Step S2302: Terminal 101 sends third information to network device 102.

[0476] In some embodiments, optional implementations of step S2302 can be found in optional implementations of step S2102 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0477] Step S2303: Network device 102 sends fourth information to terminal 101.

[0478] In some embodiments, optional implementations of step S2303 can be found in optional implementations of step S2103 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0479] Step S2304: Terminal 101 sends the fifth information to network device 102.

[0480] In some embodiments, optional implementations of step S2304 can be found in optional implementations of step S2104 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0481] Step S2305: Network device 102 sends the sixth information to terminal 101.

[0482] In some embodiments, optional implementations of step S2305 can be found in optional implementations of step S2105 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0483] Step S2306: Terminal 101 sends the sixth message to OTT server 103.

[0484] In some embodiments, optional implementations of step S2306 can be found in optional implementations of step S2107 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0485] Step S2307: OTT server 103 trains the first AI model.

[0486] In some embodiments, optional implementations of step S2307 can be found in optional implementations of step S2108 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0487] In step S2308, terminal 101 downloads the first trained AI model.

[0488] In some embodiments, optional implementations of step S2308 can be found in optional implementations of step S2109 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0489] Step S2309: Terminal 101 sends first information to network device 102.

[0490] In some embodiments, after the terminal 101 downloads the first AI model, the terminal 101 sends first information to the network device to indicate that an executable first AI model is configured on the terminal.

[0491] In step S2310, network device 102 evaluates the performance of the first AI model based on the performance requirement information of the first AI model.

[0492] In some embodiments, the performance of the first AI model is evaluated based on the performance requirement information of the first AI model, which can be achieved in at least one of the following ways:

[0493] Method 1: The network device verifies the performance of the first AI model based on the model verification requirements.

[0494] Method 2: The network device calibrates the accuracy of the first AI model based on the model accuracy requirement information.

[0495] In some embodiments, the performance of the first AI model is verified based on model validation requirement information, including at least one of the following:

[0496] a) The performance of the first AI model is verified based on the first CSI and the feedback from the first CSI.

[0497] b) Verify the performance of the first AI model based on the first CSI and the first model information; the first model information includes at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model trained by the network device, and the model structure of the first AI model trained by the network device.

[0498] c) Verify the performance of the first AI model based on the first CSI and the second model information; the second model information includes at least one of the following: the second AI model, the model parameters of the second AI model, and the model structure of the second AI model; the second AI model is the AI ​​model applied to the network device in the bilateral model.

[0499] Among them, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then compressing it into CSI.

[0500] In some embodiments, the model accuracy requirement information includes at least one of the following:

[0501] 1. First model accuracy requirement information, used to require that the accuracy of the first AI model after training is greater than a first threshold;

[0502] 2. The second model accuracy requirement information is used to require that the difference between the second CSI output by the first AI model after training and the second AI model received by the terminal is less than the second threshold.

[0503] 3. The third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the third threshold.

[0504] 4. The fourth model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the fourth threshold and greater than the fifth threshold.

[0505] 5. The fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and pairing with the second AI model received by the terminal is less than the sixth threshold and greater than the seventh threshold.

[0506] 6. The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the eighth threshold.

[0507] 7. The seventh model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the ninth threshold;

[0508] 8. The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold.

[0509] 9. The ninth model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the twelfth threshold and greater than the thirteenth threshold.

[0510] Among them, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then compressing it into CSI.

[0511] In some embodiments, optional implementations of step S2310 can refer to the optional implementation of the terminal 101 evaluating the model performance of the first AI model in step S2109 of FIG2A, and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0512] Step S2311: Network device 102 sends the eighth information to terminal 101.

[0513] In some embodiments, when the model performance of the first AI model meets the performance requirements, the network device 102 sends the eighth information to the terminal 101.

[0514] In some embodiments, optional implementations of step S2311 can be found in optional implementations of step S2111 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0515] Figure 2D is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 2D, the embodiment of the present disclosure relates to a model performance verification method, executed by a communication system 100, which includes at least one of the following steps:

[0516] Step S2401: Network device 102 sends second information to terminal 101.

[0517] In some embodiments, optional implementations of step S2401 can be found in optional implementations of step S2101 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0518] Step S2402: Terminal 101 sends third information to network device 102.

[0519] In some embodiments, optional implementations of step S2402 can be found in optional implementations of step S2102 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0520] Step S2403: Network device 102 sends fourth information to terminal 101.

[0521] In some embodiments, optional implementations of step S2403 can be found in optional implementations of step S2103 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0522] Step S2404: Terminal 101 sends the fifth information to network device 102.

[0523] In some embodiments, optional implementations of step S2404 can be found in optional implementations of step S2104 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0524] Step S2405: Network device 102 sends the sixth message to OTT server 103.

[0525] In some embodiments, optional implementations of step S2405 can be found in optional implementations of step S2205 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0526] Step S2406: OTT server 103 trains the first AI model.

[0527] In some embodiments, optional implementations of step S2406 can be found in optional implementations of step S2108 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0528] Step S2407: Terminal 101 downloads the first trained AI model.

[0529] In some embodiments, optional implementations of step S2307 can be found in optional implementations of step S2109 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0530] Step S2408: Terminal 101 sends the first information to network device 102.

[0531] In some embodiments, after the terminal 101 downloads the first AI model, the terminal 101 sends first information to the network device to indicate that an executable first AI model is configured on the terminal.

[0532] In step S2409, network device 102 evaluates the performance of the first AI model based on the performance requirement information of the first AI model.

[0533] In some embodiments, optional implementations of step S2409 can be found in optional implementations of step S2310 in FIG2C and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0534] In step S2410, network device 102 sends the eighth information to terminal 101.

[0535] In some embodiments, when the model performance of the first AI model meets the performance requirements, the network device 102 sends the eighth information to the terminal 101.

[0536] In some embodiments, optional implementations of step S2410 can be found in optional implementations of step S2111 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0537] Figure 3A is a flowchart illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 3A, the present disclosure relates to a model performance verification method, which can be executed by a terminal. The method may include:

[0538] Step S3101: Send first information to the network device; wherein, the first information is used to indicate that an executable first AI model is configured on the terminal; the first AI model is the AI ​​model applied to the terminal in the bilateral model.

[0539] In some embodiments, sending the first information to the network device can be achieved through the following steps:

[0540] The first AI model's performance met the requirements, and it sent the first message to the network device.

[0541] In some embodiments, performance requirements are used to indicate the conditions required for the model performance to achieve the desired performance.

[0542] In some embodiments, before sending the first information to the network device, the terminal also needs to perform the following steps:

[0543] Obtain the performance requirements information for the first AI model;

[0544] Based on the performance requirements, the performance of the first AI model is evaluated.

[0545] In some embodiments, the performance requirement information is used to evaluate whether the model performance of the first AI model meets the performance requirements.

[0546] In some embodiments, the performance requirements information of the first AI model can be obtained in any of the following ways:

[0547] Method 1: Receive performance requirement information sent by network devices.

[0548] Method 2: Receive performance requirement information sent by the OTT server; the performance requirement information is sent by the OTT server to the terminal after receiving the performance requirement information sent by the network device.

[0549] In some embodiments, the performance requirement information includes at least one of model validation requirement information and model accuracy requirement information.

[0550] In some embodiments, model validation requirements include information for validating the performance of a first AI model.

[0551] In some embodiments, model accuracy requirement information is used to indicate the accuracy requirement conditions under which the model performance achieves the desired performance.

[0552] The performance of an AI model can be evaluated based on performance requirements information, and this can be achieved through at least one of the following methods:

[0553] Method 1: Verify the performance of the first AI model based on the model validation requirements.

[0554] Method 2: Based on the model accuracy requirements, calibrate the accuracy of the first AI model.

[0555] In some embodiments, the first AI model is used to compress the CSI.

[0556] In some embodiments, the performance of an AI model can be verified based on model validation requirements, which can be achieved through at least one of the following methods:

[0557] Method 1: Verify the performance of the first AI model based on the first CSI and the feedback from the first CSI;

[0558] Method 2: Verify the performance of the first AI model based on the first CSI and the first model information; the first model information includes at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model trained by the network device, and the model structure of the first AI model trained by the network device;

[0559] Method 3: Verify the performance of the first AI model based on the first CSI and the second model information; the second model information includes at least one of the following: the second AI model, the model parameters of the second AI model, and the model structure of the second AI model; the second AI model is the AI ​​model applied to the network device in the bilateral model.

[0560] In some embodiments, the first CSI and the first CSI feedback are training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

[0561] In some embodiments, the model accuracy requirement information includes at least one of the following:

[0562] 1) First model accuracy requirement information, used to require that the accuracy of the first AI model after training is greater than a first threshold;

[0563] 2) Second model accuracy requirement information, which requires that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than a second threshold.

[0564] 3) The third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the third threshold.

[0565] 4) Fourth model accuracy requirement information, which requires that the difference between the second CSI output by the first AI model after training and the second AI model received by the terminal is less than the fourth threshold and greater than the fifth threshold.

[0566] 5) The fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the sixth threshold and greater than the seventh threshold.

[0567] 6) The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the eighth threshold.

[0568] 7) The seventh model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the ninth threshold;

[0569] 8) The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold.

[0570] 9) Ninth model accuracy requirement information, which requires that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the twelfth threshold and greater than the thirteenth threshold;

[0571] Among them, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then compressing it into CSI.

[0572] In some embodiments, performance requirement information is carried in any of the following:

[0573] MAC CE;

[0574] Physical layer signaling;

[0575] RRC;

[0576] User interface;

[0577] Interface between the terminal and the network node used for AI control.

[0578] In some embodiments, the first information is sent to the network device via any of the following:

[0579] MAC CE;

[0580] Physical layer signaling;

[0581] RRC;

[0582] User interface;

[0583] Interface between the terminal and the network node used for AI control;

[0584] Update Applicable Function Report; The updated Applicable Function Report is obtained by updating the Applicable Function Report, which is sent to the network device after the terminal receives the inference configuration sent by the network device; The inference configuration is sent to the terminal by the network device after the terminal receives the terminal capability query sent by the network device and sends the terminal capability report to the network device.

[0585] In some embodiments, the first AI model is obtained through any of the following methods:

[0586] Method 1: Train the first AI model based on the model data of the first AI model.

[0587] Method 2: Receive the first AI model sent by the OTT server; the first AI model is obtained by the OTT server after receiving the model data of the first AI model sent by the network device and training it based on the model data.

[0588] The model data includes at least one of the training data and model parameters of the first AI model.

[0589] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0590] Figure 3B is a flowchart illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 3B, the present disclosure relates to a model performance verification method, which can be executed by a terminal. The method may include:

[0591] Step S3201: Obtain the performance requirements information of the first AI model, which is the AI ​​model applied to the terminal in the bilateral model.

[0592] The optional implementation of step S3201 can be found in the optional implementation of step S2106 in Figure 2A, the optional implementation of step S2206 and step S2207 in Figure 2B, and other related parts in the embodiments involved in Figures 2A and 2B, which will not be repeated here.

[0593] Step S3202: Based on the performance requirement information, evaluate the performance of the first AI model.

[0594] The optional implementation of step S3202 can be found in the optional implementation of step S2110 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0595] Step S3203: The performance of the first AI model meets the performance requirements, and the first information is sent to the network device; the first information is used to indicate that an executable first AI model is configured on the terminal.

[0596] The optional implementation of step S3203 can be found in the optional implementation of step S2101 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0597] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0598] Figure 4A is a flowchart illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 4A, this embodiment of the present disclosure relates to a model performance verification method, which can be executed by a network device. The method may include:

[0599] Step S4101: Receive first information sent by the terminal; wherein, the first information is used to indicate that an executable first AI model is configured on the terminal; the first AI model is the AI ​​model applied to the terminal in the bilateral model.

[0600] In some embodiments, before the network device receives the first information sent by the terminal, the terminal has already evaluated the performance of the first AI model based on the performance requirements information of the first AI model.

[0601] In some embodiments, the performance requirement information is used to evaluate whether the model performance of the first AI model meets the performance requirements.

[0602] In some embodiments, performance requirements are used to indicate the conditions required for the model performance to achieve the desired performance.

[0603] In some embodiments, when the performance of the first AI model meets the performance requirements, the terminal sends first information to the network device. Accordingly, the network device receives the first information, indicating that the terminal has determined that the performance of the first AI model has met the performance requirements.

[0604] In some embodiments, before receiving the first information sent by the terminal, the network device also needs to perform any of the following:

[0605] Send performance requirement information to the terminal;

[0606] Send performance requirement information to the OTT server;

[0607] Among them, the performance requirements information is used to evaluate the performance of the first AI model.

[0608] In some embodiments, performance requirement information is carried in any of the following:

[0609] MAC CE;

[0610] Physical layer signaling;

[0611] RRC;

[0612] User interface;

[0613] Interface between the terminal and the network node used for AI control.

[0614] In some embodiments, the network device may also perform the following operations:

[0615] Based on the performance requirements of the first AI model, the model performance of the first AI model is evaluated.

[0616] In some embodiments, the performance requirement information includes at least one of the following:

[0617] Model validation requirements information, including relevant information for validating the performance of the first AI model;

[0618] Model accuracy requirement information indicates the accuracy requirements under which the model's performance will reach the desired performance.

[0619] In some embodiments, the first AI model is used to compress the Channel State Information (CSI).

[0620] In some embodiments, model validation requirements include at least one of the following:

[0621] First CSI and First CSI Feedback;

[0622] First CSI and first model information; the first model information includes at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model trained by the network device, and the model structure of the first AI model trained by the network device;

[0623] First CSI and second model information; second model information, including at least one of the following: second AI model, model parameters of the second AI model, and model structure of the second AI model; the second AI model is the AI ​​model applied to network devices in the bilateral model;

[0624] Among them, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then compressing it into CSI.

[0625] In some embodiments, the model accuracy requirement information includes at least one of the following:

[0626] The first model accuracy requirement information is used to require that the accuracy of the first AI model after training is greater than a first threshold.

[0627] The second model accuracy requirement information is used to require that the difference between the second CSI output by the first AI model after training and the second AI model received by the terminal is less than the second threshold.

[0628] The third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the third threshold.

[0629] The fourth model accuracy requirement information is used to require that the difference between the second CSI output by the first AI model after training and the second AI model received by the terminal is less than the fourth threshold and greater than the fifth threshold.

[0630] The fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and pairing with the second AI model received by the terminal is less than the sixth threshold and greater than the seventh threshold.

[0631] The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the eighth threshold.

[0632] The seventh model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the ninth threshold;

[0633] The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold.

[0634] The ninth model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the twelfth threshold and greater than the thirteenth threshold.

[0635] Among them, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then compressing it into CSI.

[0636] In some embodiments, the first information is received via any of the following:

[0637] MAC CE;

[0638] Physical layer signaling;

[0639] RRC;

[0640] User interface;

[0641] Interface between the terminal and the network node used for AI control;

[0642] Update the applicable function report; the updated applicable function report is obtained by updating the original applicable function report. The original applicable function report is sent to the network device after the terminal receives the inference configuration sent by the network device; the inference configuration is sent to the terminal by the network device after the terminal receives the terminal capability query sent by the network device and sends the terminal capability report to the network device.

[0643] In some embodiments, the method further includes any of the following:

[0644] Send the model data of the first AI model to the terminal;

[0645] The model data of the first AI model sent to the OTT server;

[0646] The model data includes at least one of the training data and model parameters of the first AI model.

[0647] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0648] Figure 4B is a flowchart illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 4B, the present disclosure relates to a model performance verification method, which can be executed by a network device. The method may include:

[0649] Step S4201: Receive first information sent by the terminal; wherein, the first information is used to indicate that an executable first AI model is configured on the terminal; the first AI model is the AI ​​model applied to the terminal in the bilateral model.

[0650] In some embodiments, optional implementations of step S3203 can be found in optional implementations of step S2109 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0651] In some embodiments, before the network device receives the first information sent by the terminal, the terminal only obtains the first trained AI model.

[0652] Step S4202: Based on the performance requirements information of the first AI model, evaluate the model performance of the first AI model.

[0653] In some embodiments, optional implementations of step S2402 can be found in optional implementations of step S2310 in FIG2C and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0654] In some embodiments, after the terminal sends the first information to the network device, the network device needs to evaluate the model performance of the first AI model.

[0655] Step S4203: Send the eighth message to the terminal.

[0656] In some embodiments, when the model performance of the first AI model meets the performance requirements, the network device 102 sends the eighth information to the terminal 101.

[0657] In some embodiments, optional implementations of step S2403 can be found in optional implementations of step S2111 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0658] The following describes a model performance verification method disclosed herein in further detail with reference to specific embodiments.

[0659] In some embodiments, the terminal is exemplarily a UE, and the network device is exemplarily a gNB.

[0660] In some embodiments, the server / node responsible for collecting data and training models on the base station / network side provides the UE with verification information (independent of training information) and model accuracy through the air interface Uu interface.

[0661] In some embodiments, the UE-side OTT server sends the trained model to the UE, and the UE reports updated model availability information to the base station / AI controller.

[0662] In some embodiments, the reporting method may be a new separate RRC message (e.g., model availability reporting) or reporting that the terminal has an executable model by updating the applicability reporting.

[0663] In some embodiments, the server / node on the base station / network side, which is responsible for collecting data and training models, provides verification information (independent of training information) and model accuracy to the UE-side OTT server.

[0664] In some embodiments, the UE OTT server sends the trained model to the UE, and the UE reports updated model availability information to the base station / AI controller.

[0665] In some embodiments, the reporting method may be a new separate RRC message (e.g., model availability reporting) or reporting that the terminal has an executable model by updating applicableability reporting.

[0666] In some embodiments, the verification information may include:

[0667] {target CSI, CSI feedback};

[0668] {target CSI} and the UE-side model / model parameters and model structure trained by the base station;

[0669] {target CSI} and base station model / model parameters and model structure.

[0670] In some embodiments, model accuracy may include:

[0671] The accuracy (e.g., percentage) of the model trained on the UE side is higher than a certain threshold;

[0672] The difference (average) between the CSI output by the UE-side trained model and the target CSI after pairing with the received network-side model is less than a certain threshold;

[0673] The percentage of the difference between the CSI output by the UE-side trained model and the target CSI after pairing with the received network-side model is less than a certain threshold and greater than a certain threshold;

[0674] The difference (average) between the CSI feedback obtained from the UE-side training model and the verification information CSI feedback is less than a certain threshold;

[0675] The percentage of the difference (average) between the CSI feedback and the verification information CSI feedback obtained from the UE-side training model that is less than a certain threshold is greater than a certain threshold.

[0676] In some embodiments, verification information transmission signaling and reporting model availability signaling can be performed in the following ways:

[0677] MAC CE;

[0678] Physical layer signaling;

[0679] RRC;

[0680] User interface;

[0681] Interface between the UE and network nodes used for AI control.

[0682] Figure 5A is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 5A, the embodiment of the present disclosure relates to a model performance verification method, which is executed by a communication system 100; in Figure 5A, the terminal in the communication system 100 is exemplarily a UE, and the network device is exemplarily a gNB. The above-mentioned model performance verification method includes at least one of the following steps:

[0683] Step S5101: gNB sends a UE capability query to UE.

[0684] The optional implementation of step S5101 can be found in the optional implementation of step S2101 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0685] Step S5102: The UE sends a UE capability report to the gNB.

[0686] The optional implementation of step S5102 can be found in the optional implementation of step S2102 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0687] Step S5103: gNB sends inference configuration to UE.

[0688] The optional implementation of step S5103 can be found in the optional implementation of step S2103 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0689] Step S5104: The UE sends an applicable function report to the gNB.

[0690] The optional implementation of step S5104 can be found in the optional implementation of step S2104 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0691] Step S5105: gNB sends model data to UE.

[0692] The optional implementation of step S5105 can be found in the optional implementation of step S2105 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0693] Step S5106: gNB sends the accuracy target to UE.

[0694] In some embodiments, the accuracy target and the model data in step S5105 are isolated from each other. That is, the accuracy target and the model data are carried in different signaling.

[0695] In some embodiments, optional implementations of step S5106 can be found in optional implementations of step S2106 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0696] Step S5107: The UE sends model data to the OTT server.

[0697] In some embodiments, optional implementations of step S5107 can be found in optional implementations of step S2107 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0698] Step S5108: The OTT server retrains the AI ​​model on the UE side.

[0699] In some embodiments, optional implementations of step S5108 can be found in optional implementations of step S2108 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0700] Step S5109: The UE downloads the trained AI model.

[0701] In some embodiments, optional implementations of step S5109 can be found in optional implementations of step S2109 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0702] Step S5110: The UE sends an update of applicable functions report to the gNB.

[0703] In some embodiments, optional implementations of step S5110 can be found in optional implementations of step S2110 in FIG2A and other related parts in the embodiments involved in FIG2A, which will not be repeated here.

[0704] In some embodiments, the base station (gNB) or the server / node on the network side responsible for collecting data and model training provides the UE with verification information and model accuracy through the Uu interface. The verification information and model accuracy can be understood as accuracy targets, and the verification information and model accuracy need to be independent of the training information, which can be understood as model data.

[0705] In some embodiments, the OTT server on the UE side sends the trained model (e.g., the first AI model) to the UE, and the UE reports an update of applicable functions to the base station or AI controller.

[0706] In some embodiments, the UE reports that it has an executable model by updating the applicable functionality reporting.

[0707] In some embodiments, the UE may report that it has an executable model based on a new, separate RRC message, for example, an RRC message carrying a model availability reporting.

[0708] In some embodiments, the verification information includes at least one of the following:

[0709] {target CSI, CSI feedback};

[0710] At least one of the following: {target CSI} and the UE-side model trained by the base station, model parameters, and model structure;

[0711] {target CSI} and at least one of the following: base station model, model parameters, and model structure.

[0712] In some embodiments, model accuracy may include at least one of the following:

[0713] The accuracy (e.g., percentage) of the model trained on the UE side is higher than a certain threshold;

[0714] The difference (or average) between the CSI output by the UE-side trained model and the target CSI after pairing with the received network-side model is less than a certain threshold;

[0715] The percentage of the difference (or average) between the CSI output by the UE-side trained model and the target CSI after pairing with the received network-side model is less than a certain threshold and greater than a certain threshold.

[0716] The difference (or average value) between the CSI feedback obtained from the UE-side training model and the verification information CSI feedback is less than a certain threshold;

[0717] The percentage of the difference (or average value) between the CSI feedback and the verification information CSI feedback obtained from the UE-side training model that is less than a certain threshold is greater than a certain threshold.

[0718] In some embodiments, the signaling for verifying information transmission and the signaling for reporting model suitability reports can be implemented in the following ways:

[0719] MAC CE;

[0720] Physical layer signaling;

[0721] RRC;

[0722] User interface;

[0723] Interface between the UE and network nodes used for AI control.

[0724] Step S5111: gNB sends an activation command to UE.

[0725] The model performance verification method disclosed in this embodiment may include at least one of steps S5101 to S5110. For example, step S5106 may be implemented as an independent embodiment, step S5110 may be implemented as an independent embodiment, steps S5106 and S5110 may be implemented as independent embodiments, and steps S5101, S5102, S5103, S5104, S5105, S5107, S5108, S5109, S5110 and S5111 may be implemented as independent embodiments, but are not limited thereto.

[0726] In some embodiments, the order of any two steps in steps S5101 to S5111 can be interchanged or they can be performed simultaneously.

[0727] In some embodiments, steps S5105 and S5106 may be performed in an alternate order or simultaneously.

[0728] In some embodiments, steps S5101 to S5109 and step S5111 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0729] In some embodiments, steps S5101 to S5105, S5107, S5108, S5109, S5110, and S5111 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0730] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0731] Figure 5B is an interactive schematic diagram illustrating a model performance verification method according to an embodiment of the present disclosure. As shown in Figure 5B, the embodiment of the present disclosure relates to a model performance verification method, which is executed by a communication system 100; in Figure 5B, the terminal in the communication system 100 is exemplarily a UE, and the network device is exemplarily a gNB. The above-mentioned model performance verification method includes at least one of the following steps:

[0732] Step S5201: gNB sends a UE capability query to UE.

[0733] The optional implementation of step S5201 can be found in the optional implementation of step S2101 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0734] Step S5202: The UE sends a UE capability report to the gNB.

[0735] The optional implementation of step S5202 can be found in the optional implementation of step S2102 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0736] Step S5203: gNB sends inference configuration to UE.

[0737] The optional implementation of step S5203 can be found in the optional implementation of step S2103 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0738] Step S5204: The UE sends an applicable function report to the gNB.

[0739] The optional implementation of step S5204 can be found in the optional implementation of step S2104 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0740] Step S5205: gNB sends model data to the OTT server.

[0741] The optional implementation of step S5205 can be found in the optional implementation of step S2205 in Figure 2B and other related parts in the embodiments involved in Figure 2B, which will not be repeated here.

[0742] Step S5206: gNB sends the precision target to the OTT server.

[0743] In some embodiments, the accuracy target and the model data in step S5205 are isolated from each other. That is, the accuracy target and the model data are carried in different signaling messages.

[0744] In some embodiments, optional implementations of step S5206 can be found in optional implementations of step S2206 in FIG2B and other related parts in the embodiments involved in FIG2B, which will not be repeated here.

[0745] Step S5207: The OTT server sends the accuracy target to the UE.

[0746] In some embodiments, optional implementations of step S5207 can be found in optional implementations of step S2207 in FIG2B and other related parts in the embodiments involved in FIG2B, which will not be repeated here.

[0747] Step S5208: The OTT server retrains the AI ​​model on the UE side.

[0748] In some embodiments, optional implementations of step S5208 can be found in optional implementations of step S2208 in FIG2B and other related parts in the embodiments involved in FIG2B, which will not be repeated here.

[0749] Step S5209: The UE downloads the trained AI model.

[0750] In some embodiments, optional implementations of step S5209 can be found in optional implementations of step S2109 in FIG2B and other related parts in the embodiments involved in FIG2B, which will not be repeated here.

[0751] Step S5210: The UE sends an update of applicable functions report to the gNB.

[0752] In some embodiments, optional implementations of step S5210 can be found in optional implementations of step S2210 in FIG2B and other related parts in the embodiments involved in FIG2B, which will not be repeated here.

[0753] Step S5211: gNB sends an activation command to UE.

[0754] In some embodiments, optional implementations of step S5211 can be found in optional implementations of step S2210 in FIG2B and other related parts in the embodiments involved in FIG2B, which will not be repeated here.

[0755] The model performance verification method disclosed in this embodiment may include at least one of steps S5201 to S5210. For example, step S5206 may be implemented as an independent embodiment, step S5210 may be implemented as an independent embodiment, steps S5206 and S5210 may be implemented as independent embodiments, and steps S5201, S5202, S5203, S5204, S5205, S5207, S5208, S5209, S5210 and S5211 may be implemented as independent embodiments, but are not limited thereto.

[0756] In some embodiments, the order of any two steps in steps S5201 to S5211 can be interchanged or they can be performed simultaneously.

[0757] In some embodiments, steps S5205 and S5206 may be performed in an alternate order or simultaneously.

[0758] In some embodiments, steps S5201 to S5209 and step S5211 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0759] In some embodiments, steps S5201 to S5205, S5207, S5208, S5209, S5210 and S5211 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0760] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0761] In some embodiments of this disclosure, a communication system is provided, which may include a terminal and a network device, wherein the terminal may execute the model performance verification method executed by the terminal in the foregoing embodiments of this disclosure; and the network device may execute the model performance verification method executed by the network device in the foregoing embodiments of this disclosure.

[0762] This disclosure also provides an apparatus for implementing any of the above methods. For example, an apparatus is provided that includes units or modules for implementing the steps performed by the terminal in any of the above methods. Alternatively, another apparatus is provided that includes units or modules for implementing the steps performed by a network device (e.g., an access network device, a core network functional node, a core network device, etc.) in any of the above methods.

[0763] It should be understood that the division of units or modules in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units or modules in the device can be implemented by a processor calling software: for example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of the units or modules in the above device. The processor can be, for example, a general-purpose processor, such as a Central Processing Unit (CPU) or a microprocessor, and the memory can be internal or external to the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuits. The functionality of some or all of the units or modules can be achieved through the design of these hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an Application-Specific Integrated Circuit (ASIC), and the functionality of some or all of the units or modules is achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby achieving the functionality of some or all of the units or modules. All units or modules of the above device can be implemented entirely through processor-called software, entirely through hardware circuits, or partially through processor-called software with the remaining parts implemented through hardware circuits.

[0764] In this embodiment, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a Central Processing Unit (CPU), a microprocessor, a Graphics Processing Unit (GPU) (which can be understood as a microprocessor), or a Digital Signal Processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. The logical relationships of the aforementioned hardware circuits are fixed or reconfigurable. For example, the processor is a hardware circuit implemented using an Application-Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units or modules. In addition, it can also be hardware circuits designed for artificial intelligence, which can be understood as ASICs, such as Neural Network Processing Units (NPUs), Tensor Processing Units (TPUs), and Deep Learning Processing Units (DPUs).

[0765] Figure 6A is a schematic diagram of the structure of a terminal according to an embodiment of this disclosure. As shown in Figure 6A, the terminal 101 may include at least one of a transceiver module 6101, a processing module 6102, etc. In some embodiments, the transceiver module 6101 is configured to send first information to a network device; wherein the first information is used to indicate that an executable first artificial intelligence (AI) model is configured on the terminal; the first AI model is an AI model applied to the terminal in a two-sided model. Optionally, the transceiver module 6101 can be used to perform at least one of the communication steps such as sending and / or receiving performed by the terminal 101 in any of the above methods (e.g., steps S2101 to S2107, S2111, S2112, S2201 to S2207, S2211, S2212, S2301 to S2306, S2309, S2311, S2401 to S2405, S2408, S2410, S3101, S3203, S4101, S4201, S4203, but not limited thereto), which will not be elaborated here. Optionally, the processing module 6102 can be used to execute at least one of the other steps executed by the terminal 101 in any of the above methods (e.g., steps S2108, S2109, S2210, S2208, S2209, S2210, S2307, S2308, S2310, S2406, S2407, S2409, S3201, S3203, but not limited thereto), which will not be elaborated here.

[0766] In some embodiments, the transceiver module may include a transmitting module and / or a receiving module, which may be separate or integrated. Optionally, the transceiver module may be interchangeable with a transceiver.

[0767] In some embodiments, the processing module may be a single module or may include multiple sub-modules. Optionally, the multiple sub-modules may each perform all or part of the steps required by the processing module. Optionally, the processing module may be interchangeable with a processor.

[0768] Figure 6B is a schematic diagram of the structure of a network device according to an embodiment of this disclosure. As shown in Figure 6B, the network device 102 may include at least one of a transceiver module 6201, a processing module 6202, etc. In some embodiments, the transceiver module 6201 is configured to receive first information sent by a terminal; wherein the first information is used to indicate that an executable first artificial intelligence (AI) model is configured on the terminal; the first AI model is an AI model applied to the terminal in a two-sided model. Optionally, the transceiver module 6201 can be used to perform at least one of the communication steps such as sending and / or receiving performed by the network device 102 in any of the above methods (e.g., steps S2101 to S2107, S2111, S2112, S2201 to S2207, S2211, S2212, S2301 to S2306, S2309, S2311, S2401 to S2405, S2408, S2410, S3101, S3203, S4101, S4201, S4203, but not limited thereto), which will not be elaborated here. Optionally, the processing module 6202 may be used to execute at least one of the other steps executed by the network device 102 in any of the above methods (e.g., steps S2108, S2109, S2210, S2208, S2209, S2210, S2307, S2308, S2310, S2406, S2407, S2409, S3201, S3203, but not limited thereto), which will not be elaborated here.

[0769] In some embodiments, the transceiver module may include a transmitting module and / or a receiving module, which may be separate or integrated. Optionally, the transceiver module may be interchangeable with a transceiver.

[0770] In some embodiments, the processing module may be a single module or may include multiple sub-modules. Optionally, the multiple sub-modules may each perform all or part of the steps required by the processing module. Optionally, the processing module may be interchangeable with a processor.

[0771] Figure 7A is a schematic diagram of the structure of the communication device 7100 proposed in an embodiment of this disclosure. The communication device 7100 can be a network device (e.g., access network device, core network device, etc.), a terminal (e.g., user equipment, etc.), a chip, chip system, or processor that supports the first device in implementing any of the above methods, or a chip, chip system, or processor that supports the terminal in implementing any of the above methods. The communication device 7100 can be used to implement the methods described in the above method embodiments; for details, please refer to the descriptions in the above method embodiments.

[0772] As shown in Figure 7A, the communication device 7100 includes one or more processors 7101. The processor 7101 can be a general-purpose processor or a dedicated processor, such as a baseband processor or a central processing unit (CPU). The baseband processor can be used to process communication protocols and communication data, while the CPU can be used to control communication devices (e.g., base stations, baseband chips, IoT devices, IoT device chips, DUs or CUs, etc.), execute programs, and process program data. The communication device 7100 is used to execute any of the above methods.

[0773] In some embodiments, the communication device 7100 further includes one or more memories 7102 for storing instructions. Optionally, all or part of the memories 7102 may also be located outside the communication device 7100.

[0774] In some embodiments, the communication device 7100 further includes one or more transceivers 7103. When the communication device 7100 includes one or more transceivers 7103, the transceivers 7103 perform communication steps such as sending and / or receiving in the above method (e.g., steps S2101-S2107, S2111, S2112, S2201-S2207, S2211, S2212, S2301-S2306, S2309, S2311, S2401-S2405, S2408, S2410, S31). At least one of steps S3203, S4101, S4201, and S4203 (but not limited thereto), processor 7101 executes at least one of other steps (e.g., steps S2108, S2109, S2210, S2208, S2209, S2210, S2307, S2308, S2310, S2406, S2407, S2409, S3201, and S3203, but not limited thereto).

[0775] In some embodiments, a transceiver may include a receiver and / or a transmitter, which may be separate or integrated. Optionally, the terms transceiver, transceiver unit, transceiver, transceiver circuit, etc., may be used interchangeably; the terms transmitter, transmitting unit, transmitter, transmitting circuit, etc., may be used interchangeably; and the terms receiver, receiving unit, receiver, receiving circuit, etc., may be used interchangeably.

[0776] In some embodiments, the communication device 7100 may include one or more interface circuits. Optionally, the interface circuit is connected to the memory 7102, and the interface circuit can be used to receive signals from the memory 7102 or other devices, and can be used to send signals to the memory 7102 or other devices. For example, the interface circuit can read instructions stored in the memory 7102 and send the instructions to the processor 7101.

[0777] The communication device 7100 described in the above embodiments may be a first device or an Internet of Things (IoT) device, but the scope of the communication device 7100 described in this disclosure is not limited thereto, and the structure of the communication device 7100 may not be limited by FIG. 7A. The communication device may be a standalone device or part of a larger device. For example, the communication device may be: (1) a standalone integrated circuit IC, or chip, or chip system or subsystem; (2) a collection of one or more ICs, optionally, the IC collection may also include storage components for storing data and programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, IoT device, smart IoT device, cellular phone, wireless device, handheld device, mobile unit, vehicle device, first device, cloud device, artificial intelligence device, etc.; (6) others, etc.

[0778] Figure 7B is a schematic diagram of the structure of the chip 7200 according to an embodiment of this disclosure. For cases where the communication device 7100 can be a chip or a chip system, the schematic diagram of the chip 7200 shown in Figure 7B can be referenced, but is not limited thereto.

[0779] Chip 7200 includes one or more processors 7201, which are used to perform any of the above methods.

[0780] In some embodiments, chip 7200 further includes one or more interface circuits 7203. Optionally, interface circuit 7203 is connected to memory 7202, and interface circuit 7203 can be used to receive signals from memory 7202 or other devices, and interface circuit 7203 can be used to send signals to memory 7202 or other devices. For example, interface circuit 7203 can read instructions stored in memory 7202 and send the instructions to processor 7201.

[0781] In some embodiments, the interface circuit 7203 performs communication steps such as sending and / or receiving in the above method (e.g., steps S2101-S2107, S2111, S2112, S2201-S2207, S2211, S2212, S2301-S2306, S2309, S2311, S2401-S2405, S2408, S2410, S3101, S32). 03. At least one of steps S4101, S4201, and S4203, but not limited thereto, the processor 7201 executes at least one of other steps (e.g., steps S2108, S2109, S2210, S2208, S2209, S2210, S2307, S2308, S2310, S2406, S2407, S2409, S3201, and S3203, but not limited thereto).

[0782] In some embodiments, the terms interface circuit, interface, transceiver pin, transceiver, etc., can be used interchangeably.

[0783] In some embodiments, chip 7200 further includes one or more memories 7202 for storing instructions. Optionally, all or part of the memories 7202 may be located outside of chip 7200.

[0784] This disclosure also proposes a storage medium storing instructions that, when executed on a communication device 7100, cause the communication device 7100 to perform any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but not limited thereto; it may also be a storage medium readable by other devices. Optionally, the storage medium may be a non-transitory storage medium, but not limited thereto; it may also be a temporary storage medium.

[0785] This disclosure also provides a program product that, when executed by the communication device 7100, causes the communication device 7100 to perform any of the above methods. Optionally, the program product may be a computer program product.

[0786] This disclosure also proposes a computer program that, when run on a computer, causes the computer to perform any of the above methods.

Claims

1. A model performance verification method, executed by a terminal, characterized in that, The method includes: Send the first message to the network device; The first information is used to indicate that the terminal is configured with an executable first artificial intelligence (AI) model; the first AI model is the AI ​​model applied to the terminal in the bilateral model.

2. The method according to claim 1, characterized in that, Sending the first information to the network device includes: Once the performance of the first AI model meets the performance requirements, the first information is sent to the network device; the performance requirements are used to indicate the conditions required for the model performance to reach the expected performance.

3. The method according to claim 1 or 2, characterized in that, Before sending the first information to the network device, the method further includes: Obtain the performance requirement information of the first AI model; the performance requirement information is used to evaluate whether the model performance of the first AI model meets the performance requirements, and the performance requirements are used to indicate the required conditions for the model performance to reach the expected performance. Based on the performance requirements information, the performance of the first AI model is evaluated.

4. The method according to claim 3, characterized in that, The step of obtaining the performance requirement information of the first AI model includes any one of the following: Receive the performance requirement information sent by the network device; The performance requirement information is received from the over-the-top OTT server; the performance requirement information is sent to the terminal by the OTT server after receiving the performance requirement information sent by the network device.

5. The method according to claim 3 or 4, characterized in that, The performance requirement information includes at least one of model validation requirement information and model accuracy requirement information; the model validation requirement information includes relevant information for validating the performance of the first AI model; the model accuracy requirement information indicates the accuracy requirement conditions for the model performance to reach the expected performance. The evaluation of the AI ​​model's performance based on the performance requirement information includes at least one of the following: Based on the model verification requirements, the performance of the first AI model is verified; Based on the model accuracy requirement information, the accuracy of the first AI model is calibrated.

6. The method according to claim 5, characterized in that, The verification of the AI ​​model's performance based on the model verification requirement information includes at least one of the following: The performance of the first AI model is verified based on the first channel state information (CSI) and the first CSI feedback. Based on the first CSI and the first model information, the performance of the first AI model is verified; the first model information includes at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model trained by the network device, and the model structure of the first AI model trained by the network device. Based on the first CSI and the second model information, the performance of the first AI model is verified; the second model information includes at least one of the following: the second AI model, the model parameters of the second AI model, and the model structure of the second AI model; the second AI model is the AI ​​model applied to the network device in the bilateral model; Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

7. The method according to claim 5 or 6, characterized in that, The model accuracy requirements include at least one of the following: The first model accuracy requirement information is used to require that the accuracy of the first AI model after training is greater than a first threshold. The second model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than a second threshold. The third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the third threshold. The fourth model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the fourth threshold and greater than the fifth threshold. The fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the sixth threshold and greater than the seventh threshold. The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the eighth threshold. The seventh model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the ninth threshold. The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold. The ninth model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the twelfth threshold and greater than the thirteenth threshold; Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

8. The method according to any one of claims 3-7, characterized in that, The performance requirement information is carried in any of the following: MAC control element MAC CE; Physical layer signaling; Radio Resource Control (RRC); User interface; The interface between the terminal and the network node used for AI control.

9. The method according to any one of claims 1-8, characterized in that, The first AI model is used to compress CSI.

10. The method according to any one of claims 1-9, characterized in that, The first information is sent to the network device via any of the following: MAC CE; Physical layer signaling; RRC; User interface; The interface between the terminal and the network node used for AI control; Update the applicable function report; the updated applicable function report is obtained by updating the applicable function report, which is sent to the network device after the terminal receives the inference configuration sent by the network device; the inference configuration is sent to the terminal by the network device after the terminal receives the terminal capability query sent by the network device and sends the terminal capability report to the network device.

11. The method according to any one of claims 1-10, characterized in that, The first AI model is obtained through any of the following methods: The first AI model is trained based on the model data of the first AI model; The first AI model is received from the OTT server; the first AI model is obtained by the OTT server after receiving the model data of the first AI model sent by the network device, and training it based on the model data. The model data includes at least one of the training data and model parameters of the first AI model.

12. A model performance verification method, executed by a network device, characterized in that, The method includes: The first message sent by the receiving terminal; The first information is used to indicate that the terminal is configured with an executable first artificial intelligence (AI) model; the first AI model is the AI ​​model applied to the terminal in a two-sided model.

13. The method according to claim 12, characterized in that, Before the first information sent by the receiving terminal, the method further includes any one of the following: Send performance requirement information to the terminal; Send performance requirement information to the OTT server; The performance requirement information is used to evaluate whether the performance of the first AI model meets the performance requirements, and the performance requirements are used to indicate the conditions required for the model performance to reach the expected performance.

14. The method according to claim 13, characterized in that, The performance requirement information is contained in any of the following: MAC control element MAC CE; Physical layer signaling; Radio Resource Control (RRC); User interface; The interface between the terminal and the network node used for AI control.

15. The method according to claim 12, characterized in that, The method further includes: Based on the performance requirements of the first AI model, the model performance of the first AI model is evaluated. The performance requirement information is used to evaluate whether the performance of the first AI model meets the performance requirements, and the performance requirements are used to indicate the conditions required for the model performance to reach the expected performance.

16. The method according to any one of claims 13-15, characterized in that, The performance requirements information includes at least one of the following: Model validation requirements information, including relevant information for validating the performance of the first AI model; Model accuracy requirement information indicates the accuracy requirements under which the model's performance reaches the desired performance.

17. The method according to claim 16, characterized in that, The model validation requirements include at least one of the following: First Channel State Information (CSI) and First CSI Feedback; First CSI and first model information; the first model information includes at least one of the following: the first AI model trained by the network device, the model parameters of the first AI model trained by the network device, and the model structure of the first AI model trained by the network device; First CSI and second model information; the second model information includes at least one of the following: second AI model, model parameters of the second AI model, and model structure of the second AI model; the second AI model is the AI ​​model applied to the network device in the bilateral model; Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

18. The method according to claim 16 or 17, characterized in that, The model accuracy requirements include at least one of the following: The first model accuracy requirement information is used to require that the accuracy of the first AI model after training is greater than a first threshold. The second model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than a second threshold. The third model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the third threshold. The fourth model accuracy requirement information is used to require that the difference between the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the fourth threshold and greater than the fifth threshold. The fifth model accuracy requirement information is used to require that the average value of the second CSI and the first CSI output by the first AI model after training and the second AI model received by the terminal is less than the sixth threshold and greater than the seventh threshold. The sixth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the eighth threshold. The seventh model accuracy requirement information is used to require that the average of the second CSI feedback and the first CSI feedback obtained by the first AI model after training be less than the ninth threshold. The eighth model accuracy requirement information is used to require that the difference between the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the tenth threshold and greater than the eleventh threshold. The ninth model accuracy requirement information is used to require that the average value of the second CSI feedback and the first CSI feedback obtained by the first AI model after training is less than the twelfth threshold and greater than the thirteenth threshold; Wherein, the first CSI and the first CSI feedback are the training sample data of the first AI model, and the first CSI feedback is obtained by inputting the first CSI into the first AI model and then performing CSI compression.

19. The method according to any one of claims 12-18, characterized in that, The first AI model is used to compress CSI.

20. The method according to any one of claims 12-19, characterized in that, The first information is received via any of the following: MAC CE; Physical layer signaling; RRC; User interface; The interface between the terminal and the network node used for AI control; The updated applicable function report is obtained by updating the original applicable function report, which is sent to the network device by the terminal after receiving the inference configuration sent by the network device; the inference configuration is sent to the terminal by the network device after the terminal receives the terminal capability query sent by the network device and sends the terminal capability report to the network device.

21. The method according to any one of claims 1-20, characterized in that, The method further includes any one of the following: Send the model data of the first AI model to the terminal; The model data of the first AI model sent to the OTT server; The model data includes at least one of the training data and model parameters of the first AI model.

22. A communication method, characterized in that, The method includes: The terminal sends first information to the network device; wherein the first information is used to indicate that the terminal is configured with an executable first artificial intelligence (AI) model; the first AI model is the AI ​​model applied to the terminal in a two-sided model; The network device receives the first information sent by the terminal.

23. A communication device, characterized in that, include: One or more processors; The communication device is used to execute the model performance verification method according to any one of claims 1-11 or the model performance verification method according to any one of claims 12-21.

24. A communication device, characterized in that, The communication device is used to execute the model performance verification method according to any one of claims 1-11 or the model performance verification method according to any one of claims 12-21.

25. A communication system, characterized in that, The device includes a terminal and a network device, wherein the terminal is configured to implement the model performance verification method according to any one of claims 1-11, and the network device is configured to implement the model performance verification method according to any one of claims 12-21.

26. A storage medium storing instructions, characterized in that, When the instruction is executed on the communication device, the communication device performs the model performance verification method as described in any one of claims 1-11, or performs the model performance verification method as described in any one of claims 12-21.

27. A program product comprising at least one of a program and instructions, characterized in that, When at least one of the programs or instructions is executed by a communication device, it implements the model performance verification method according to any one of claims 1-11, or implements the steps of the model performance verification method according to any one of claims 12-21.

Images