Ai model monitoring method and related device

Abstract

Provided in the present application is an AI model monitoring method. A terminal device requests a first network device to monitor the performance of an AI model deployed by itself, comprising: the terminal device sending a first request message to the first network device; after receiving the first request message sent by the terminal device, the first network device determining a first test sample for testing the performance of the AI model, a position label that corresponds to the first test sample being located in a valid region that corresponds to the AI model, thus effectively testing the performance of the AI model; sending a first instruction to the terminal device to instruct the terminal device to use the first test sample to test the AI model, so as to obtain a positioning result; the terminal device sending a second request message to the first network device to request the first network device to determine whether the positioning result output by the AI model is trustworthy; on the basis of a positioning error, the first network device determining whether the AI model is trustworthy, so as to obtain a determination result; and returning the determination result to the terminal device.

Description

An AI model monitoring method and related equipment

[0001] This application claims priority to Chinese Patent Application No. 2025101295268, filed on January 27, 2025, entitled "An AI Model Monitoring Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of wireless communication technology, and in particular to an AI model monitoring method and related equipment. Background Technology

[0003] In New Radio (NR) or Long Term Evolution (LTE) systems, user equipment (UE) mobility management, such as UE location, is typically based on base stations. Accurate location is crucial for UE mobility management. Artificial Intelligence (AI) has been introduced into wireless communication networks and can be applied to UE location scenarios.

[0004] Typically, AI models deployed on the UE side are trained using training sets in a specific wireless environment. When the wireless environment changes or the training set coverage is incomplete, the positioning results of the AI ​​model may not be reliable. Therefore, it is necessary to monitor whether the AI ​​model deployed on the UE side is adapted to the specific UE to ensure the accuracy of the positioning results based on the AI ​​model. Summary of the Invention

[0005] This application provides an AI model monitoring method and related equipment to monitor the AI ​​model and ensure the accuracy of the positioning results.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] This application provides an AI model monitoring method applicable to a first network device. The first network device generates an AI model with positioning capabilities using training samples and distributes the AI ​​model to a terminal device, which then uses the AI ​​model for positioning. Specifically, the terminal device sends a first request message to the first network device, requesting the first network device to monitor the AI ​​model. Upon receiving the first request message, the first network device determines a first test sample located within an effective area from a test sample set based on the first request message, and sends a first instruction to the terminal device, instructing the terminal device to perform positioning based on the AI ​​model and the first test sample to obtain a positioning result. The effective area refers to the area where the terminal device is located, where the AI ​​model can be used for positioning. The test sample set includes at least the test sample determined based on the sample sent by the terminal device, and the confidence level of the location label corresponding to the test sample is greater than a confidence threshold, thereby ensuring the accuracy of the test. After receiving the first instruction, the terminal device performs positioning based on the AI ​​model and the first test sample, obtains the positioning result, and sends a second request message to the first network device, requesting the first network device to determine whether the positioning result output by the AI ​​model is reliable. The first network device obtains the positioning error, and based on the positioning error, obtains a judgment result on whether the AI ​​model is reliable, and sends the judgment result to the terminal device.

[0008] In this application, to enable the AI ​​model deployed on the UE side to adapt to specific terminal devices, the terminal device can request a first network device to monitor the performance of its deployed AI model; that is, the terminal device sends a first request message to the first network device. Upon receiving the first request message from the terminal device, the first network device determines a first test sample for testing the AI ​​model's performance. The location tag corresponding to this first test sample is located within the valid area corresponding to the AI ​​model, thus enabling effective testing of the AI ​​model's performance. A first instruction is sent to the terminal device, instructing it to test the AI ​​model using the first test sample to obtain positioning results. The terminal device then sends a second request message to the first network device, requesting the first network device to determine whether the positioning results output by the AI ​​model are reliable. The first network device determines the reliability of the AI ​​model based on the positioning error, obtains a judgment result, and returns the judgment result to the terminal device. The positioning error is determined based on the positioning result and the location tag of the first test sample.

[0009] In this way, since the first test sample is determined based on the sample sent by the terminal device, it reflects the signal measurement capability of the terminal device. Therefore, when testing the performance of the AI ​​model using the first test sample, the test results can reflect the compatibility between the AI ​​model and the terminal device. Furthermore, when the AI ​​model is detected as unreliable, it indicates that the compatibility between the AI ​​model and the terminal device is poor. This allows for fine-tuning or retraining of the AI ​​model to better match the signal measurement capability of the terminal device, thereby improving the accuracy of the AI ​​model's localization.

[0010] Furthermore, the confidence level of the location label corresponding to the first test sample is greater than the confidence level threshold, thereby ensuring the effectiveness and accuracy of testing the performance of the AI ​​model using the first test sample and improving the accuracy of monitoring.

[0011] The content of the first instruction varies depending on the application scenario. If the terminal device stores the first test sample and its location tag, the first instruction only needs to carry the sample identifier of the first test sample. The terminal device then uses the sample identifier in the first instruction to determine the first test sample and test the AI ​​model. In this scenario, the first network device only needs to transmit the sample identifier to the terminal device, reducing transmission overhead.

[0012] If the terminal device stores a first test sample but lacks a location tag, the first instruction can carry the sample identifier of the first test sample and the location tag corresponding to that first test sample. Alternatively, in this scenario, if the terminal device does not need to determine the positioning error based on the positioning result and the location tag, the first instruction can also carry only the sample identifier of the first sample, without carrying the location tag.

[0013] If the terminal device can delete the sample after uploading it to the first network device to reduce its own storage space usage, in this scenario, the first instruction includes the first test sample. If the terminal device needs to determine the positioning error, the first instruction may also include the location tag corresponding to the first test sample.

[0014] In some implementations, if the positioning error is determined by the terminal device, the second request message includes the positioning error based on the positioning result and the location tag corresponding to the first test sample.

[0015] In some implementations, if the positioning error is determined by the first network device, the second request message includes the positioning result. After receiving the second request message, the first network device determines the positioning error based on the positioning result and the location label of the first test sample.

[0016] The first network device can determine whether the positioning error exceeds a threshold. If it does, the AI ​​model is deemed unreliable; otherwise, it is considered reliable. However, many factors can cause an AI model to be unreliable, such as changes in the wireless environment or incomplete training sample coverage. Different factors require different optimization strategies. Therefore, when determining whether an AI model is unreliable, the factors causing this unreliability can also be identified for targeted optimization.

[0017] Specifically, if the positioning error is greater than the error threshold, the distribution difference between the test sample set and the training sample set is determined; if the distribution difference is less than the difference threshold, the factor causing the AI ​​model to be unreliable is determined as the first factor; otherwise, the factor causing the AI ​​model to be unreliable is determined as the second factor. The judgment result may include the factor causing the AI ​​model to be unreliable.

[0018] Specifically, when the distribution difference is less than the difference threshold, it indicates that the distributions of the training samples and the test samples are similar, and the wireless environment remains unchanged. In this case, the first factor is a model error, and the terminal device should stop using the AI ​​model and retrain it. When the distribution difference is greater than the difference threshold, it indicates that there is a significant difference in the distributions between the training samples and the test samples, and the wireless environment has changed. In this case, the second factor is model aging, and the terminal device should stop using the AI ​​model and retrain it.

[0019] Regarding the first factor, the first network device will retrain the AI ​​model based on the test sample set and the training sample set to obtain a new AI model, and then distribute the new AI model to the terminal device. Regarding the second factor, the first network device will retrain the AI ​​model based on the test sample set and the target training sample set to obtain a new AI model, and then distribute the new AI model to the terminal device.

[0020] The target training sample set is a subset of the training sample set, and the regions indicated by the location labels of the training samples in the target training sample set do not overlap with the regions indicated by the location labels of the test samples in the test sample set.

[0021] In some implementations, if the positioning error is less than an error threshold, it indicates that the AI ​​model is reliable. In this case, it can be further determined whether a target test sample set exists. If it does, it indicates that incremental training of the AI ​​model is required. The target test sample set is a subset of the test sample set, and the regions indicated by the location labels of the test samples in the target test sample set do not overlap with the regions indicated by the location labels of the training samples in the training sample set.

[0022] If a target test sample set is identified, the judgment result may include an instruction to train the AI ​​model. In this case, the first network device can use the target test sample set to train the AI ​​model and send the trained AI model to the terminal device, thereby enabling monitoring and optimization of the AI ​​model's performance.

[0023] In some implementations, to ensure that the first test sample selected belongs to the test sample within the valid area, the first request message includes the identification information of the valid area corresponding to the AI ​​model. Then, the first network device selects the first test sample located in the valid area from the test sample set based on the location label of the test sample.

[0024] In some implementations, the first network device can determine the first test sample from the test sample set by first determining whether a target test sample exists in the test sample set or whether a target training sample exists in the training sample set. If so, it determines an intersection region based on the position labels of the test samples in the test sample set and the position labels of the training samples in the training sample set, and determines the test sample located in the intersection region as the first test sample. The position labels corresponding to the target test sample and the target training sample do not belong to the intersection region.

[0025] In some implementations, if the test sample set also includes a second test sample, the first instruction further includes the second test sample. This first instruction instructs the terminal device to perform positioning based on the AI ​​model and the second test sample to obtain a positioning result. The second test sample is a test sample located in the valid area, determined by the first network device based on the second sample set sent by the second network device. The first network device can be a Location Management Function (LMF) network element, and the second network device can be a Positioning Reference Unit (PRU).

[0026] If the terminal device needs to determine the positioning error, the first instruction may also include the location tag corresponding to the second test sample.

[0027] To monitor the performance of the AI ​​model, it is necessary to obtain the test samples used for monitoring. The test sample set on the first network device side is obtained by the terminal device upon request. Specifically, the terminal device sends a third request message to the first network device, which requests the first network device to obtain the test samples for monitoring the AI ​​model. The first network device obtains the test sample set and sends a notification message to the terminal device to notify the terminal device that it has obtained the test samples for monitoring the AI ​​model.

[0028] The acquisition of the test sample set includes: after receiving a third request message, the first network device sends a second instruction to the terminal device, which instructs the terminal device to collect first target data. The data type of the first target data includes the data type required by the terminal device for localization using the AI ​​model. After receiving the second instruction, the terminal device collects the first target data and determines a first sample set based on the first target data, then sends the first sample set to the first network device. The first network device identifies the first sample in the first sample set with a confidence level greater than a confidence threshold as a test sample. The confidence level indicates the credibility of the location label corresponding to the first sample.

[0029] In this implementation, the first network device obtains test samples from the terminal device side, thereby determining whether the AI ​​model matches the terminal device when monitoring the AI ​​model. Furthermore, the location tags of the test samples have high reliability, ensuring effective monitoring of the AI ​​model.

[0030] In some implementations, if the terminal device can determine the location label of the first sample, the first sample set will always include the location label corresponding to the first sample and the confidence level corresponding to the location label.

[0031] If the terminal device cannot accurately determine the location label of the first sample, then the first sample set will only include the first sample. In this case, after receiving the first sample set, the first network device will determine the location label and confidence level corresponding to the first sample based on the positioning algorithm.

[0032] The first target data may include measurement data, timestamps of the acquired measurement data, and configuration information. The measurement data may include cellular measurement data, and the configuration information may include at least one of the following: acquisition start and end time, and acquisition frequency. Specifically, the cellular measurement data includes cell identifiers and reference signal measurement characteristics. Further, it may also include time difference of arrival and angle of arrival.

[0033] In some implementations, the measurement data may also include one or more of the following: global navigation satellite system measurement data, wireless fidelity measurement data, Bluetooth measurement data, and sensor measurement data.

[0034] Specifically, GNSS policy data may include location tags and confidence levels; wireless fidelity measurement data and Bluetooth measurement data may include wireless access point identifiers and received signal strength, respectively; sensor measurement data may include measurement data acquired based on at least one of the following sensors: accelerometer, gyroscope, and magnetometer.

[0035] In some implementations, the location of the second network device is typically fixed. To increase the reliability and diversity of the test samples, the first network device sends a third instruction to the second network device, which instructs the second network device to collect second target data. The data type of the second target data includes at least the data types required for localization using the AI ​​model. After receiving the third instruction, the second network device collects the second target data and determines a second sample set based on the second target data, then sends this second sample set to the first network device. The first network device then designates each second sample in the second sample set as a test sample.

[0036] The second target data includes cellular measurement data, location tags, and timestamps. The specific format of the cellular measurement data can be found in the description above.

[0037] In some implementations, if the third request message sent by the terminal device to the first network device includes the identification information of the effective area corresponding to the AI ​​model, then the location label corresponding to the first sample in the first sample set uploaded by the terminal device to the first network device is located within the effective area.

[0038] Based on this, the third instruction may include the identification information of the valid area, and the location label of the second sample in the second sample set uploaded by the second network device is located in the valid area.

[0039] The identification information of the effective area corresponding to the AI ​​model includes the location information of the effective area and / or a list of cell identifiers within the effective area.

[0040] In some implementations, due to the configuration of the terminal device, the terminal device may disagree with the collection of the first target data. Based on this, the terminal device can send a feedback message to the first network device to request the first network device to reconfigure the target data to be collected. Upon receiving the feedback message, the first network device determines the third target data to be collected based on the feedback message and sends a fourth instruction to the terminal device. This fourth instruction instructs the terminal device to collect the third target data. The third target data differs from the first target data, and the data type indication in the third target data includes data type identification using an AI model.

[0041] The feedback message can include reasons for disagreeing with the collection of the primary target data, such as resource limitations or user privacy settings.

[0042] Secondly, this application provides an AI model monitoring method applied to a terminal device equipped with an AI model having positioning capabilities. Specifically, the method involves: sending a first request message to a first network device, requesting monitoring of the AI ​​model; receiving a first instruction from the first network device instructing the terminal device to perform positioning using the AI ​​model, the first instruction instructing the terminal device to perform positioning based on the AI ​​model and a first test sample. The location tag corresponding to the first test sample is located within the valid area corresponding to the AI ​​model, and the first test sample is determined by the first network device from a set of test samples based on the first request message. This set of test samples includes at least test samples determined based on samples sent by the terminal device, and the confidence level of the test samples is greater than a confidence threshold; performing positioning based on the AI ​​model and the first test sample to obtain a positioning result; sending a second request message to the first network device, requesting a judgment on the reliability of the positioning result output by the AI ​​model; and receiving a judgment result from the first network device regarding the reliability of the AI ​​model. The judgment result is determined by the first network device based on a positioning error, which is determined based on the positioning result and the location tag corresponding to the first test sample.

[0043] In some implementations, the first instruction includes a sample identifier of the first test sample. Alternatively, the first instruction includes a sample identifier of the first test sample and a location label corresponding to the first test sample. Or, the first instruction includes the first test sample and a location label corresponding to the first test sample.

[0044] In some implementations, the terminal device can determine the positioning error based on the positioning result and the location tag corresponding to the first test sample. The second request message includes the positioning error.

[0045] In some implementations, if the positioning error is determined by the first network device, the second request message includes the positioning result. The positioning error is determined by the first network device based on the positioning result and the location tag corresponding to the first test sample.

[0046] In some implementations, if the judgment result indicates that the AI ​​model is untrustworthy, the judgment result may include factors that cause the AI ​​model to be untrustworthy; if the judgment result indicates that the AI ​​model is trustworthy, the judgment result may also include an instruction to train the AI ​​model.

[0047] In some implementations, if the AI ​​model is retrained by the first network device, the terminal device receives the retrained AI model sent by the first network device.

[0048] In some implementations, if the AI ​​model is retrained by the terminal device, the following methods may be included:

[0049] If the unreliability of the AI ​​model is the primary factor, obtain a set of test samples and a set of training samples, and retrain the AI ​​model based on the set of test samples and the set of training samples.

[0050] If the unreliability of the AI ​​model is considered a second factor, a test sample set and a target training sample set are obtained, and the AI ​​model is retrained based on these two sets. The target training sample set is a subset of the training sample set, and the regions indicated by the location labels of the training samples in the target training sample set do not overlap with the regions indicated by the location labels of the test samples in the test sample set.

[0051] In some implementations, if the judgment result indicates that the AI ​​model is trustworthy, a target test sample set is obtained, and the AI ​​model is trained using the target test sample set. The target test sample set is a subset of the test sample set, and the regions indicated by the location labels of the test samples in the target test sample set do not overlap with the regions indicated by the location labels of the training samples in the training sample set.

[0052] In some implementations, the first request message includes identification information of the valid region corresponding to the AI ​​model, and the first request message is used to request the monitoring of the AI ​​model using test samples within the valid region.

[0053] In some implementations, if the test sample set also includes a second test sample, the first instruction also includes the second test sample, which is determined by the first network device based on the second sample set sent by the second network device.

[0054] In some embodiments, the method further includes: the terminal device sending a third request message to the first network device, the third request message being used to request the first network device to obtain test samples of the monitoring AI model; and receiving a notification message sent by the first network device, the notification message being used to notify that test samples of the monitoring AI model have been obtained.

[0055] In some embodiments, the method further includes: receiving a second instruction sent by a first network device, the second instruction being used to instruct a terminal device to collect first target data, the data type of the first target data including at least the data type required by the terminal device for positioning using an AI model; acquiring the first target data, and determining a first sample set based on the first target data, the first sample set including a first sample and a sample identifier corresponding to the first sample; and sending the first sample set to the first network device.

[0056] In some implementations, the first sample set may also include the location label corresponding to the first sample and the confidence level corresponding to the location label.

[0057] In some implementations, the first target data includes measurement data, timestamps of the measurement data acquisition, and configuration information.

[0058] In some implementations, the measurement data includes cellular measurement data, and the configuration information includes at least one of the following: start and end time of data collection, and collection frequency. Cellular measurement data includes cell identifiers and reference signal measurement characteristics.

[0059] In some implementations, the measurement data may also include one or more of the following: Global Navigation Satellite System measurement data, Wireless Fidelity measurement data, Bluetooth measurement data, and sensor measurement data.

[0060] In some implementations, GNSS measurement data includes location tags and confidence levels; Wi-Fi and Bluetooth measurement data include wireless access point (AP) identifiers and received signal strength (RSSI), respectively; and sensor measurement data includes measurement data acquired based on at least one of an accelerometer, gyroscope, and magnetometer.

[0061] In some embodiments, the method further includes: if the terminal device does not agree to collect the first target data, sending a feedback message to the first network device, the feedback message requesting the first network device to reconfigure the target data to be collected; and receiving a fourth instruction sent by the first network device, the fourth instruction being used to instruct the terminal device to collect the third target data.

[0062] In some implementations, the feedback message carries the reason for disagreeing with the collection of the first target data.

[0063] Thirdly, an AI model monitoring method is provided. This method is applied to a second network device, whose location is fixed, for assisting in localization; specifically, it can be a Localization Reference Unit (PRU). The method includes: receiving a third instruction sent by a first network device, the third instruction instructing the second network device to collect second target data, the data type of which includes at least the data types required for localization using the AI ​​model; acquiring the second target data and determining a second sample set based on the second target data; and sending the second sample set to the first network device, so that the first network device determines test samples based on the second sample set and uses the test samples to monitor the AI ​​model.

[0064] The second sample set includes a second sample, a sample identifier corresponding to the second sample, and a location label corresponding to the second sample. The location label indicates the location of the second network device.

[0065] In some implementations, the second target data includes cellular measurement data, location tags, and timestamps.

[0066] In some implementations, the second instruction includes identification information of the effective region corresponding to the AI ​​model, and the second sample refers to a sample located within the effective region.

[0067] Fourthly, this application provides a network element, which includes a transceiver and a processor; wherein the transceiver is used to perform the receiving operation and the transmitting operation in the method described in the first aspect or any embodiment of the first aspect; and the processor is used to perform other operations in the method described in the first aspect or any embodiment of the first aspect besides the receiving operation and the transmitting operation.

[0068] In some embodiments, the transceiver is configured to perform the receiving and transmitting operations in the method described in the third aspect or any embodiment of the third aspect; the processor is configured to perform other operations in the method described in the third aspect or any embodiment of the third aspect besides the receiving and transmitting operations.

[0069] Fifthly, this application provides a terminal device, including a transceiver and a processor; wherein the transceiver is used to perform the receiving operation and the transmitting operation in the method described in the second aspect or any embodiment of the second aspect; and the processor is used to perform other operations in the method described in the second aspect or any embodiment of the second aspect besides the receiving operation and the transmitting operation.

[0070] Sixthly, this application provides a communication system including a terminal device and a network element. The network element is configured to execute the method described in the first aspect or any embodiment thereof, or to execute the method described in the third aspect or any embodiment thereof; the terminal device is configured to execute the method described in the second aspect or any embodiment thereof.

[0071] In a seventh aspect, this application provides a computer storage medium for storing a computer program, which, when executed, implements the AI ​​model monitoring method provided in any one of the first to third aspects of this application.

[0072] Eighthly, this application provides a computer program product containing instructions that, when run on at least one computing device, causes the at least one computing device to implement the AI ​​model monitoring method provided in any one of the first to third aspects of this application. Attached Figure Description

[0073] Figure 1 is a structural diagram of an exemplary communication system provided in this application;

[0074] Figure 2 is a schematic flowchart of a method for obtaining test samples provided in this application;

[0075] Figure 3 is a schematic diagram of another method for obtaining test samples provided in this application;

[0076] Figure 4 is an interactive schematic diagram of an AI model monitoring method provided in an embodiment of this application;

[0077] Figure 5 is a schematic diagram of a region division provided in an embodiment of this application;

[0078] Figure 6 is an interactive schematic diagram of another AI model monitoring method provided in an embodiment of this application;

[0079] Figure 7 is a schematic diagram of the structure of a network element provided in this application;

[0080] Figure 8 is a structural schematic diagram of a terminal device provided in this application. Detailed Implementation

[0081] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0082] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0083] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0084] Currently, AI has been introduced into wireless communication networks and is widely used in many application scenarios of air interface technology. One common application scenario is the location of terminal devices based on AI models. When locating terminal devices based on AI models, AI nodes can be deployed in one or more of the following: network devices, terminal devices, core networks, and positioning devices; or, AI nodes can be deployed independently, such as in a location other than any of the above-mentioned devices.

[0085] AI nodes are used to perform AI-related operations. For example, AI-related operations may include one or more of the following: model failure testing, model performance testing, model training testing, and data collection. For instance, a network device can forward AI model-related data reported by a terminal device to an AI node, which then performs the AI-related operations. Alternatively, a network device or terminal device can forward AI model-related data to an AI node, which then performs the AI-related operations. Another example is that an AI node can send one or more of the outputs of AI-related operations, such as a trained neural network model, model evaluation, or test results, to a network device and / or a terminal device. Optionally, the AI ​​node can directly send the outputs of AI-related operations to a network device and a terminal device. Yet another example is that an AI node can send the outputs of AI-related operations to a terminal device via a network device. And yet another example is that an AI node can send the outputs of AI-related operations to a network device via a terminal device.

[0086] In current solutions, the location management function (LMF) network element in the core network can perform the training of the AI ​​model and then distribute the trained AI model to the terminal device. The terminal device uses positioning reference signals (PRS) received from one or more base stations as input to the AI ​​model, and the output of the AI ​​model is the terminal device's location information. Typically, the LMF generates training samples based on specific environmental parameters when training the AI ​​model. Therefore, when using the AI ​​model for positioning, the same environmental parameters should be used to ensure the accuracy of the location predicted by the AI ​​model.

[0087] However, due to changes in the wireless environment, different terminal devices having different signal measurement capabilities, or incomplete coverage of training samples, the positioning results of terminal devices based on AI models may not be reliable.

[0088] Based on this, this application provides an AI model monitoring method, in which a terminal device can request a first network device to monitor the performance of its deployed AI model. Upon receiving a first request message, the first network device identifies a first test sample for performance testing of the AI ​​model and sends a first instruction to the terminal device. This instruction instructs the terminal device to perform localization based on the AI ​​model and the first test sample, obtaining a localization result. After obtaining the localization result, the terminal device sends a second request message to the first network device, requesting the first network device to determine whether the localization result output by the AI ​​model is reliable. The first network device determines the reliability of the AI ​​model based on the localization error (the error between the localization result and the location label corresponding to the first test sample) and sends the determination result to the terminal device, thereby monitoring the localization accuracy of the AI ​​model.

[0089] The first test sample is determined based on the signals actually collected by the terminal device. This first test sample can reflect the signal measurement capability of the terminal device, so that the judgment result can reflect whether the AI ​​model is compatible with the terminal device. In order to make adaptive adjustments to the AI ​​model based on the judgment result, the accuracy of the terminal device's positioning based on the AI ​​model can be improved.

[0090] The terminal devices in this application include various devices with wireless communication capabilities, which can be used to connect people, objects, machines, etc. These terminal devices can be widely applied in various scenarios, such as: cellular communication, D2D, V2X, peer-to-peer (P2P), M2M, MTC, IoT, virtual reality (VR), augmented reality (AR), industrial control, autonomous driving, telemedicine, smart grids, smart furniture, smart offices, smart wearables, smart transportation, smart cities, drones, robots, remote sensing, passive sensing, positioning, navigation and tracking, autonomous delivery, and other scenarios.

[0091] Terminal devices can be terminals in any of the above scenarios, such as MTC terminals, IoT terminals, etc. Terminal devices can be user equipment (UE), terminals, fixed equipment, mobile station equipment or mobile devices, subscriber units, handheld devices, vehicle-mounted equipment, wearable devices, cellular phones, smartphones, session initialization protocol (SIP) phones, wireless data cards, personal digital assistants (PDAs), computers, tablets, laptops, wireless modems, handsets, laptop computers, computers with wireless transceiver capabilities, smart books, vehicles, satellites, global positioning system (GPS) devices, target tracking devices, aircraft (e.g., drones, helicopters, multiple helicopters, four helicopters, or airplanes), ships, remote control devices, smart home devices, industrial equipment, or devices built into the above devices (e.g., communication modules, modems, or chips in the above devices), or other processing devices connected to a wireless modem. For ease of description, the terminal device will be described below using the terminal or UE as an example.

[0092] It should be understood that in certain scenarios, a UE can also be used as a base station. For example, a UE can act as a scheduling entity, providing sidelink signaling between UEs in scenarios such as V2X, D2D, or P2P.

[0093] In this embodiment, the device for implementing the functions of the terminal device can be the terminal device itself, or it can be a device that supports the terminal device in implementing the functions, such as a chip system or a chip. This device can be installed in the terminal device. In this embodiment, the chip system can be composed of chips, or it can include chips and other discrete devices.

[0094] The network device in this application embodiment can be a device for communicating with a terminal device. This network device can also be called an access network device or a wireless access network device, such as a base station. In this application embodiment, the network device can refer to a radio access network (RAN) node (or device) that connects the terminal device to the wireless network. Base stations can broadly encompass various names like those listed below, or be interchangeable with them, such as: NodeB, evolved NodeB (eNB), next-generation NodeB (gNB), relay station, access point, transmission reception point (TRP), transmitting point (TP), master station, auxiliary station, motor slide retainer (MSR) node, home base station, network controller, access node, wireless node, access point (AP), transmission node, transceiver node, baseband unit (BBU), remote radio unit (RRU), active antenna unit (AAU), remote radio head (RRH), central unit (CU), distributed unit (DU), positioning node, etc.

[0095] A base station can be a macro base station, micro base station, relay node, donor node, or a combination thereof. A base station can also refer to a communication module, modem, or chip installed within the aforementioned equipment or apparatus. A base station can also be a mobile switching center and equipment performing base station functions in D2D, V2X, and M2M communications, network-side equipment in 6G networks, and equipment performing base station functions in future communication systems. A base station can support networks using the same or different access technologies. The embodiments of this application do not limit the specific technologies or equipment forms used in the network equipment.

[0096] Base stations can be fixed or mobile. For example, a helicopter or drone can be configured to act as a mobile base station, and one or more cells can move depending on the location of the mobile base station. In other examples, a helicopter or drone can be configured as a device to communicate with another base station.

[0097] In this embodiment, the apparatus for implementing the function of the network device can be the network device itself, or it can be an apparatus capable of supporting the network device in implementing that function, such as a chip system or a chip. This apparatus can be installed within the network device. In this embodiment, the chip system can be composed of chips, or it can include chips and other discrete components.

[0098] Network devices and terminal devices can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; they can also be deployed on water; and they can also be deployed in the air on airplanes, balloons, and satellites. This application does not limit the scenario in which the network devices and terminal devices are located. Furthermore, terminal devices and network devices can be hardware devices, software functions running on dedicated hardware, or software functions running on general-purpose hardware, such as virtualization functions instantiated on a platform (e.g., a cloud platform), or entities that include dedicated or general-purpose hardware devices and software functions. This application does not limit the specific form of the terminal devices and network devices.

[0099] Referring to Figure 1, this figure is a schematic diagram of the communication system provided in this application. The wireless communication system includes an access network 100 and a core network 200. Optionally, the communication system may also include the Internet 300. The wireless access network 100 may be a next-generation (e.g., 6G or higher) wireless access network, or a traditional (e.g., 5G, 4G, 3G or 2G) wireless access network.

[0100] One or more terminal devices 120 (only one is shown in the figure) can be interconnected or connected to one or more network devices in the access network 100. In Figure 1, the access network 100 includes network devices 110a and 110b as an example.

[0101] Figure 1 is for illustrative purposes only. The wireless communication system may also include other devices, such as core network equipment, wireless relay equipment, and / or wireless backhaul equipment. In practical applications, this wireless communication system can include multiple network devices (also called access network devices) and multiple terminal devices simultaneously, without limitation. A network device can serve one or more terminal devices simultaneously. A terminal device can also access one or more network devices simultaneously.

[0102] To monitor AI models, it's necessary to first obtain a set of test samples with location tags, and then use these test samples to test the AI ​​model's performance, thereby determining the reliability of the AI ​​model deployed on a specific terminal device. The following section will introduce how to obtain a set of test samples with location tags, using a specific implementation example.

[0103] Referring to Figure 2, which is an interactive diagram of a method for obtaining test samples provided in an embodiment of this application, in this example, the terminal device is UE and the first network device is LMF, specifically including:

[0104] S201: The UE sends a third request message to the LMF, which requests the LMF to obtain test samples of the monitoring AI model.

[0105] In this embodiment, since the AI ​​model is deployed on the UE side, in order to ensure the accuracy of the positioning results, the UE can actively send a third request message to the LMF to request the LMF to obtain test samples with location tags, so as to monitor the performance of the AI ​​model through the test samples.

[0106] Specifically, the UE can periodically send a third request message to the LMF, or send a third request message to the LMF when it detects a change in the signal distribution, or when the signal distribution changes significantly. A change in signal distribution may be due to a change in the wireless environment, such as a change in network configuration by the operator; or it may be due to a change in the UE's signal measurement capabilities, such as an update to the UE's hardware or software.

[0107] S202: The LMF sends a second instruction to the UE, which instructs the UE to collect the first target data.

[0108] In this embodiment, after receiving the third request message, the LMF determines the first target data that the UE needs to collect. The LMF can send a second instruction to the UE via the LTE positioning protocol (LPP).

[0109] The first target data may include, but is not limited to, one or more measurement data, timestamps for the collected measurement data, and configuration information. The purpose of the timestamps is to treat measurement data collected at the same time as a sample. The configuration information includes at least the start and end times of the data collection, the collection frequency, and the measurement conditions. The start and end times of the data collection are related to the frequency of the monitoring AI model and can be triggered periodically or by the UE itself.

[0110] Specifically, the measurement data may include one or more of the following: cellular measurement data, Global Navigation Satellite System (GNSS) measurement data, Wi-Fi measurement data, Bluetooth measurement data, and sensor measurement data.

[0111] The cellular measurement data may include cell ID and reference signal measurement characteristics. Specifically, it may also include one or more of the angle of arrival (AoA) and time difference of arrival (TDOA) relative to the base station.

[0112] Among them, the cell identifier can be the Earfcn candidate index (ECI), the cell global identifier (CGI), the physical cell identities (PCI), or the absolute radio frequency channel number (ARFCN).

[0113] Specifically, the reference signal measurement characteristics include one or more of the following: reference signal received power (RSRP), reference signal received quality (RSRQ), and signal-to-interference plus noise ratio (SINR). Specifically, the UE receives a positioning reference signal (PRS) transmitted by the base station, which may carry the aforementioned Cell ID, RSRP, RSRQ, SINR, AOA, and TDOA.

[0114] GNSS measurement data includes real location tags and their reliability, used to determine the UE's location; Wi-Fi and Bluetooth measurement data include unique access point (AP) identifiers and received signal strength indicators (RSSI), used to assist in obtaining the UE's real (approximate) location; sensor measurement data is optional, including measurement data obtained from accelerometers, gyroscopes, magnetometers, etc., used to assist in obtaining the UE's real (approximate) location.

[0115] Additionally, the first target data may also include a location tag and a corresponding confidence level for that location tag. The location tag indicates the location of the UE, and the confidence level indicates the reliability of the location tag. However, the first target data collected by the UE may not carry a location tag. For example, when the UE is indoors, it cannot initiate GNSS positioning, making it impossible to determine the UE's true location.

[0116] It should be noted that, to ensure the data collected by the UE can be used for testing the AI ​​model, the data types included in the first target data must be consistent with the data types included in the training samples used to train the AI ​​model. In other words, the data types in the first target data must at least include the data types required by the terminal device for positioning using the AI ​​model. For example, if the training samples for training the AI ​​model include cell identifiers, reference signal measurement features, and TDOA, then the first target data collected by the UE will also include cell identifiers, reference signal measurement features, and TDOA. As another example, if the training samples for training the AI ​​model include cell identifiers, reference signal measurement features, TDOA, and AOA, then the first target data collected by the UE will also include cell identifiers, reference signal measurement features, TDOA, and AOA.

[0117] S203: The UE acquires the first target data and determines the first sample set based on the first target data.

[0118] S204: The UE sends the first sample set to the LMF.

[0119] In this embodiment, after receiving a first instruction, the UE collects first target data based on the first instruction. The UE can collect the aforementioned measurement data at different times and then combine different measurement data collected at the same timestamp as a single sample to obtain a first sample set. For example, cellular measurement data, GNSS measurement data, Wi-Fi measurement data, Bluetooth measurement data, and sensor data collected at time t1 can be used as first sample 1; cellular measurement data, GNSS measurement data, Wi-Fi measurement data, Bluetooth measurement data, and sensor data collected at time t2 can be used as first sample 2; and cellular measurement data, GNSS measurement data, Wi-Fi measurement data, Bluetooth measurement data, and sensor data collected at time t3 can be used as first sample 3. After determining the first sample set, the UE can send the first sample set to the LMF via the LPP protocol.

[0120] Specifically, when the UE can determine its true location based on GNSS, the samples in the first sample set may carry location tags; if the UE cannot determine its true location based on GNSS, the samples in the first sample set may not carry location tags. It should be noted that if the UE is moving during the acquisition of the first target data, the location tags corresponding to different samples in the first sample set may be different; if the UE is stationary, the location tags corresponding to different samples in the first sample set may be the same.

[0121] When the UE determines the first sample set, it can assign a sample identifier to each first sample. The first sample set also includes the sample identifier corresponding to each first sample, so that the unique first sample can be determined through the sample identifier.

[0122] S205: LMF determines the first sample in the first sample set with a confidence level greater than the confidence threshold as the test sample.

[0123] Upon receiving a first sample set, if the samples in the first sample set carry location tags and confidence levels, the LMF can directly select the first sample with a confidence level greater than a confidence threshold as a test sample. If the samples in the first sample set do not carry location tags, the LMF first determines the location tags and confidence levels of the samples using traditional localization algorithms, and then selects the first sample with a high confidence level as a test sample. The confidence threshold is determined based on the actual application scenario, and this embodiment does not impose a limitation on it.

[0124] For example, LMF uses parameters such as TDOA / AOA / Round-Trip Time (RTT) to determine location labels and confidence levels using RAT-dependent positioning methods, or based on positioning methods such as Wi-Fi positioning and Bluetooth positioning. Alternatively, in scenarios with continuous time-series data acquisition, LMF can use sensors, vehicle dead-reckoning (VDR), and pedestrian dead-reckoning (PDR) algorithms to determine location labels and confidence levels.

[0125] To further increase the diversity of test samples and improve monitoring accuracy, the LMF can also send a third command to other network devices, instructing them to collect second target data. Specifically, these other network devices refer to devices capable of assisting in positioning, such as Location Reference Units (PRUs). PRUs can be divided into static PRUs and dynamic PRUs. Static PRUs have fixed locations, typically deployed in specific locations, providing stable positioning references; dynamic PRUs have variable locations. In this embodiment, to obtain more reliable location labels, samples collected by static PRUs will be used simultaneously to monitor the AI ​​model on the UE side.

[0126] It should be noted that, considering different UEs may have different signal measurement capabilities in the same wireless environment, and the AI ​​model trained by LMF is usually general-purpose, to determine whether the AI ​​model deployed on a UE is suitable for that UE, it is necessary to monitor the reliability of the AI ​​model based on the data collected from that UE. Furthermore, since the signal measurement capabilities of the PRU and the UE differ, and the location of the static PRU is fixed, it can provide test samples with more reliable locations, increasing the diversity of test samples and improving the accuracy of monitoring.

[0127] Based on this, the above method also includes the following steps, as shown in Figure 3:

[0128] S301: The LMF sends a third instruction to the PRU, which instructs the PRU to collect data from the second target.

[0129] In this embodiment, after receiving the third request message sent by the UE, the LMF can simultaneously send a second instruction to the UE and a third instruction to the PRU, so as to instruct the PRU to collect the second target data through the third instruction.

[0130] The second target data may include cellular measurement data, location tags, and timestamps. The specific content of the cellular measurement data can be found in the relevant descriptions above. Specifically, the data types in the second target data include at least the data types required for input when using the AI ​​model for localization. The timestamps are used to group different types of data collected at the same time as a single sample. Each sample typically has a location tag, which indicates the location of the PRU (Programmer, Receiver, and Unit).

[0131] It should be noted that the execution order of S202 and S301 is not subject to the above restrictions. S202 can be executed first and then S301; or S301 can be executed first and then S202; or S202 and S301 can be executed simultaneously.

[0132] S302: PRU acquires the second target data and determines the second sample set based on the second target data.

[0133] S303: PRU sends the second sample set to LMF.

[0134] After receiving the third instruction, the PRU begins collecting data from the second target. Based on the timestamp of the collected data, it identifies data corresponding to the same timestamp as a second sample, thus obtaining a second sample set, which is then sent to the LMF. The second sample set includes the second sample, its corresponding sample identifier, and its corresponding location tag. The location tag indicates the PRU's deployment location.

[0135] S304: LMF determines the test samples from each of the second samples in the second sample set.

[0136] Since the location indicated by the location label corresponding to the second sample is the actual location of the PRU, each second sample in the second sample set can be identified as a test sample, thereby using the test sample to monitor the AI ​​model.

[0137] Typically, the applicable cells for an AI model are fixed, and different AI models can be generated for different cells. That is, each AI model corresponds to a usable effective area, and the positioning results of a UE located within that effective area based on that AI model are reliable. Since the UE may be in a mobile state, to ensure that the collected data can be applied to the currently deployed AI model, data located within the effective area needs to be collected. Therefore, the third request message sent by the UE to the LMF can include the identifier of the effective area corresponding to the AI ​​model. The third instruction sent by the LMF to the PRU includes the identifier of this effective area, thereby instructing the PRU that the second sample collected belongs to the effective area.

[0138] The identifier for the valid area can be its location information, i.e., the coordinates of its geographical boundary (Validity Area), or it can be a list of cell identifiers (AreaID-CellList) within the valid area. It should be noted that if training samples are not collected for some areas within the valid area during AI model training, the valid area is larger than the area formed by the location labels corresponding to the training samples.

[0139] Specifically, both the UE and PRU can determine whether a sample belongs to the valid area when collecting samples using the following methods: If the sample carries a location tag, it can be compared with the geographical boundary coordinates of the valid area to determine whether the sample is located within the valid area; if the sample does not carry a location tag, it is determined whether the cell identifier in the cellular measurement data corresponding to the sample is included in the cell identifier list. If it is included, the sample is determined to be located within the valid area; otherwise, the sample is determined not to be located within the valid area.

[0140] In some application scenarios, due to UE device status, resource limitations, user privacy settings, etc., the UE may be unable to collect the first target data indicated by the second instruction. In this case, the UE needs to request the LMF to reconfigure the data to be collected. Specifically, after receiving the second instruction, if the terminal device detects that it does not agree to collect the first target data for some reason, the UE sends a feedback message to the LMF, which requests the LMF to reconfigure the target data to be collected. The LMF determines the third target data to be collected based on the feedback message and sends a fourth instruction to the UE, which instructs the UE to collect the third target data.

[0141] The third target data differs from the first target data. This difference can be determined based on the reason why the terminal disagrees with the collection of the first target data. Therefore, the feedback message can carry the reason for disagreement. Specifically, this difference can be reflected in the different data types in the third target data compared to the first target data. For example, the first target data includes both cellular measurement data and GNSS measurement data, while the third target data only includes cellular measurement data.

[0142] Furthermore, when it is necessary to rely on the PRU to collect test samples, the LMF will also send a fifth instruction to the PRU after reconfiguring the data to be collected, instructing the PRU to recollect new target data through the fifth instruction.

[0143] S305: The LMF sends a notification message to the UE, which informs the UE that test samples have been obtained for monitoring the AI ​​model.

[0144] In this embodiment, after receiving a notification message, and when the UE is in a state where AI model monitoring is acceptable, it can request the LMF to monitor the AI ​​model. The state where the UE is in an acceptable monitoring state may include, but is not limited to, situations where the time interval since the last monitoring exceeds a time threshold or the UE is in an idle state.

[0145] It should be noted that Figure 3 is only used as an example to understand the acquisition of test samples. In actual applications, the execution order of S301-S304 and the execution order of S202-S205 are not subject to the above restrictions. By executing S205 and S304, the set of test samples required to monitor the AI ​​model can be determined.

[0146] To facilitate understanding of the specific implementation of AI model monitoring, the following will be explained in conjunction with specific embodiments.

[0147] Referring to Figure 4, which is an interactive diagram of an AI model monitoring method provided in an embodiment of this application, the method includes:

[0148] S401: The UE sends a first request message to the LMF, which is used to request the monitoring of the AI ​​model.

[0149] In this embodiment, the UE sends a first request message to the LMF via the LPP protocol in order to monitor the AI ​​model with positioning function through the first request message.

[0150] S402: The LMF determines a first test sample from the test sample set based on the first request message, and the location label corresponding to the first test sample is located within the valid area.

[0151] The test sample set includes at least the test sample determined by LMF based on the first sample uploaded by the UE, and the confidence level of the location tag corresponding to the test sample is greater than the confidence level threshold.

[0152] In this embodiment, in order to improve the accuracy of monitoring the AI ​​model, it is necessary to ensure that the location label corresponding to the determined first test sample is located in the effective area corresponding to the AI ​​model, so that the AI ​​model can make accurate predictions about the first test sample.

[0153] Furthermore, to improve testing accuracy, a first test sample whose location label falls within the region corresponding to the training sample can be selected from the test sample set. Specifically, when determining the first test sample, it is first determined whether a target test sample or a target training sample exists between the test sample set and the training sample set. If so, the intersection region is determined based on the location labels of the test samples in the test sample set and the location labels of the training samples in the training sample set; the test sample located within the intersection region is then identified as the first test sample. Note that the AI ​​model is generated based on the training sample set, and the location labels corresponding to both the target test sample and the target training sample do not belong to the intersection region.

[0154] To facilitate understanding of the aforementioned intersection region, target test sample, and target training sample, please refer to the sample partitioning diagram shown in Figure 5. The intersection region and unique region are determined by comparing the position labels of test sample set A with the position labels of training sample set B, which can be achieved in the following way.

[0155] (1) The region is gridded and each grid is encoded.

[0156] (2) Check if there are test samples in each grid of set A. If there are, mark the grid as A_TRUE, otherwise mark it as A_FALSE. Similarly, mark the grid of set B as B_TRUE or B_False.

[0157] (3) Treat the grids that are simultaneously marked as A_True and B_True as intersection grids, merge them into an intersection region, and label all samples (including samples from A and B) in this region as R_ABinter (as shown in the black area of ​​Figure 5); treat the grids that are simultaneously marked as A_True and B_False as grids unique to set A, merge them into a region unique to set A, and label the samples in A in this region as R_Auniq (as shown in the white area of ​​Figure 5); treat the grids that are simultaneously marked as A_False and B_True as grids unique to set B, merge them into a region unique to set B, and label the samples in B in this region as R_Buniq (as shown in the underlined area of ​​Figure 5).

[0158] (4) Select the samples with tag R_ABinter from sets A and B respectively as sample sets A_inter and B_inter. Select the samples with tag R_Auniq from set A as test samples A_unique. Select the samples with tag R_Buniq from set B as training sample set B_unique.

[0159] The first test sample refers to the sample in the sample set A_inter.

[0160] To ensure that the location label of the first test sample determined by LMF is located within the effective area of ​​the AI ​​model, the first request message may include the identification information of the effective area of ​​the AI ​​model. Then, LMF will select the first test sample located within the effective area from the test sample set based on the location label of the test sample.

[0161] In its implementation, LMF can first filter out a set of test samples A that are located within the valid region based on the location labels in the test sample set, so that the location labels corresponding to the samples in the sample set A_inter are all located within the valid region.

[0162] It should be noted that if the UE uploads the first sample set of the valid area during the test sample set generation stage, then the above filtering operation does not need to be performed again during the monitoring stage.

[0163] S403: The LMF sends a first instruction to the UE, which instructs the UE to locate based on the AI ​​model and the first test sample.

[0164] In different application scenarios, the content carried by the first instruction can be different. When the test sample set only includes the first sample uploaded by the UE, and the uploaded first sample carries a location tag, since the UE can store the first sample it has collected, the first instruction can carry the sample identifier of the first test sample. Thus, after receiving the first instruction, the UE can obtain the first test sample based on the sample identifier in the first instruction. This can reduce air interface transmission overhead.

[0165] When the test sample set only includes the first sample uploaded by the UE, and the uploaded first sample does not have a location tag, the first instruction can carry the sample identifier and location tag of the first test sample, so that the UE can determine the positioning error based on the location tag when performing positioning tests on the AI ​​model based on the first test sample. If the UE does not need to determine the positioning error, the first instruction can also carry only the sample identifier of the first test sample.

[0166] After the UE sends the first sample set to the LMF, it can delete the first sample set to reduce the occupation of local storage resources. In this case, the first instruction includes the first test sample and the location tag corresponding to the first test sample.

[0167] Furthermore, when the test sample set also includes a second sample uploaded by the PRU, the test samples located in the valid area also include the second test sample. Since the UE does not store the second test sample, the first instruction can also include the second test sample. Thus, the first instruction can instruct the UE to perform a positioning test on the AI ​​model using the second test sample. If the UE needs to determine the positioning error, the first instruction also includes the location tag corresponding to the second test sample, so that the UE can determine the positioning error based on the positioning result output by the AI ​​model and the location tag corresponding to the second test sample.

[0168] S404: The UE performs localization based on the AI ​​model and the first test sample to obtain the localization result.

[0169] S405: The UE sends a second request message to the LMF, which requests a determination of whether the positioning result output by the AI ​​model is reliable.

[0170] After receiving the first instruction, the UE uses the first test sample to perform a positioning test on the AI ​​model and obtains the positioning result output by the AI ​​model. After obtaining the positioning result, the UE sends a second request message to the LMF to request the LMF to determine the reliability of the positioning result.

[0171] S406: LMF obtains the judgment result of whether the AI ​​model is reliable based on the positioning error.

[0172] In this embodiment, after receiving the second request message, LMF obtains the positioning error and determines whether the AI ​​model is trustworthy based on the positioning error, and obtains the judgment result.

[0173] The positioning error can be determined by the UE or by the LMF, and the specific implementation can be determined according to the actual application. If the positioning error is determined by the UE, after obtaining the positioning result, the UE determines the positioning error based on the positioning result and the location label of the first test sample, and sends the positioning error to the LMF through the second request message.

[0174] If the positioning error is determined by the LMF, the UE sends the positioning result to the LMF via a second request message. The LMF then determines the positioning error based on the positioning result and the location tag of the first test sample. In this way, the UE only needs to send the positioning result to the LMF and does not need to perform the operation of calculating the positioning error, thus reducing the overhead on the UE side.

[0175] The specific form of positioning error can be determined based on the form of the location labels. Specifically, if the location labels are 2D / 3D location coordinates, mean squared error (MSE), root mean squared error (RMSE), or mean absolute error (MAE) can be used as the measurement index of positioning error; if the location labels are grid indexes, precision, recall, or accuracy can be used as the measurement index of positioning error.

[0176] Specifically, S406 can be implemented in the following way: determine whether the positioning error is less than the error threshold. If it is less than the threshold, it indicates that the positioning result is accurate and the AI ​​model is reliable. If it is greater than or equal to the error threshold, it indicates that the positioning result is inaccurate and the AI ​​model is unreliable.

[0177] S407: The LMF sends the judgment result to the UE.

[0178] Upon receiving the judgment result, if the judgment result indicates that the AI ​​model is untrustworthy, the UE will stop using the AI ​​model. Meanwhile, considering that the LMF stores test and training sample sets, the UE can send an update request to the LMF to request optimization or update of the AI ​​model. Alternatively, the UE can trigger optimization or update of the AI ​​model itself when it determines that the AI ​​model is untrustworthy. Since the UE does not store the test and training samples collected by the PRU, the LMF will send the test and training samples collected by the PRU to the UE, and the UE will perform the AI ​​model update or optimization operation.

[0179] Typically, when an AI model is deemed unreliable, it is optimized and trained using test samples. However, in some scenarios, the unreliability of an AI model may stem from flawed training algorithms or parameter design, rendering even retraining with test samples ineffective. To address this issue, this embodiment provides a solution: first, determine the cause of the AI ​​model's unreliability, and then implement effective measures to optimize the model based on that specific cause. Specifically, the distribution difference between the test sample set and the training sample set is assessed. If the difference is less than a threshold, it indicates similarity, and the factor causing the unreliability of the AI ​​model is identified as the first factor, with the assessment result including this first factor. Otherwise, it indicates a significant difference in distribution, and the factor causing the unreliability of the AI ​​model is identified as the second factor, with the assessment result including this second factor.

[0180] Distribution difference refers to the difference in data distribution between test samples and training samples. Specifically, it can manifest as different cell identifiers for test samples and training samples, or different values ​​of reference signal measurement features in test samples and training samples. Distribution difference may be caused by changes in network configuration (e.g., changing the signal transmission power) or changes in the UE's own signal measurement capabilities.

[0181] Specifically, distributional tests, such as the multivariate Kolmogorov-Smirnov test, Mann-Whitney U test, and MANOVA, can be used to examine the distributional differences between the test sample set and the training sample set. It should be noted that if the first test sample refers to the test sample located within the intersection region, then when determining the distributional differences, the distributional differences between the test samples and the training samples within the intersection region will be considered.

[0182] The first factor refers to internal factors such as incomplete training set coverage or incorrect model training (e.g., unreasonable training algorithm or parameter configuration), which lead to unreliable AI models. The second factor refers to external factors such as changes in the wireless environment, which lead to AI model aging.

[0183] Specifically, if the unreliability of the AI ​​model is the primary factor, then the AI ​​model should be retrained using both the test and training sample sets. That is, if the wireless environment remains unchanged but the model is flawed, the training algorithm or parameter configuration needs to be modified, and the model should be retrained using both the test and training sample sets.

[0184] If the unreliability of the AI ​​model is the second factor, then the AI ​​model is retrained based on the test sample set and the target training sample set. That is, when the wireless environment changes, indicating that the model has aged, the AI ​​model is retrained using the test sample set and the target training sample set. Specifically, the regions indicated by the location labels of the training samples in the target training sample set and the regions indicated by the location labels of the test samples in the test sample set do not overlap; that is, the target training sample set is B_unique in Figure 5.

[0185] Furthermore, when the AI ​​model is reliable, it can be further determined whether a target test sample set exists in the test sample set. If it does, it is determined that the AI ​​model needs to be incrementally trained using the target test sample set. In this case, the determination result also includes an instruction to perform incremental training on the AI ​​model. The target test sample set is a subset of the test sample set, and the regions indicated by the position labels of the test samples in the target test sample set do not overlap with the regions indicated by the position labels of the training samples in the training sample set. That is, the target test sample set is A_unique in Figure 5.

[0186] If the AI ​​model optimization operation is performed by the LMF, the LMF will fine-tune the AI ​​model using the target test sample set and then send the fine-tuned AI model to the UE.

[0187] If the UE performs the AI ​​model optimization operation, the LMF can send the target test sample set to the UE, and then the UE can use the target test sample set to fine-tune the AI ​​model.

[0188] Referring to Figure 6, which illustrates an interactive diagram of an AI model monitoring method, the difference between this method and Figure 4 lies in the following steps:

[0189] S601: The UE sends a fourth request message to the LMF to request the samples needed to update the AI ​​model.

[0190] In this embodiment, after receiving the judgment result, if the judgment result indicates that the AI ​​model needs to be updated, the UE sends a fourth request message to the LMF.

[0191] S602: LMF determines the samples needed to update the AI ​​model based on the fourth request message.

[0192] S603: The LMF sends the samples needed to update the AI ​​model to the UE.

[0193] In this embodiment, since the LMF performs the operation of determining whether the AI ​​model is trustworthy and indicating whether the AI ​​model needs to be updated, the LMF can determine the samples required to update the AI ​​model after receiving the fourth request message when the AI ​​model needs to be updated.

[0194] For example, if the first factor causes the AI ​​model to be unreliable, then the samples required to update the AI ​​model include both the test sample set and the training sample set. If the second factor causes the AI ​​model to be unreliable, then the samples required to update the AI ​​model include both the test sample set and the target training sample set. If the judgment indicates that the AI ​​model is reliable and needs to be updated, then the samples required to update the AI ​​model include the target test sample set.

[0195] S604: The UE uses the received samples to train the AI ​​model.

[0196] After acquiring a sample, the UE uses that sample to train the AI ​​model. Once a new AI model is acquired, the UE uses the new AI model to replace the deactivated AI model, thus completing the monitoring and updating of the AI ​​model's performance.

[0197] It should be noted that when performing AI model updates on the UE side, the LMF needs to transmit samples to the UE side, resulting in significant air interface overhead. To reduce air interface overhead, AI model updates are typically performed on the LMF side.

[0198] As can be seen, by using the above scheme, when it is determined that the AI ​​model is unreliable, the specific reasons for the unreliability can be analyzed, and then corresponding optimization strategies can be implemented based on the specific reasons, so that the optimized AI model can be adapted to the UE and improve the positioning accuracy.

[0199] The hardware implementation methods of network elements and terminal devices will be further introduced below with reference to Figures 7 and 8.

[0200] Referring to Figure 7, a schematic diagram of the hardware structure of a network element is shown. This network element can be used to execute the methods performed by the LMF in the embodiments shown in Figures 2, 3, 4, and 6. The network element shown in Figure 7 includes at least one processor 111, at least one memory 112, at least one transceiver 113, at least one network interface 114, and one or more antennas 115. The processor 111, memory 112, transceiver 113, and network interface 114 are connected, for example, through a bus. In this embodiment, the connection may include various interfaces, transmission lines, or buses, etc., and this embodiment is not limited to these. The antenna 115 is connected to the transceiver 113. The network interface 114 is used to enable the network element to connect with other communication devices through a communication link. For example, the network interface 114 may include a network interface between the network element and network elements in the core network, such as an S1 interface. The network interface may also include a network interface between the network element and other network elements, such as an X2 or Xn interface.

[0201] Specifically, the processor 111 shown in Figure 4 can perform the network element processing actions in the above method, the memory 112 can perform the storage actions in the above method, the transceiver 113 and the antenna 115 can perform the air interface transmission and reception actions in the above method, and the network interface 114 can perform the interaction actions with network elements or other network elements in the above method.

[0202] The processor in this application embodiment, such as processor 111, may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may form a SoC (System-on-a-Chip) with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits), or it may be integrated as a built-in processor in an ASIC. The ASIC with the integrated processor may be packaged separately or packaged together with other circuits. In addition to including cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or logic circuits that implement dedicated logic operations.

[0203] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.

[0204] The memory 112 can exist independently and be connected to the processor 111. Optionally, the memory 112 can be integrated with the processor 111, for example, integrated into a single chip. The memory 112 can store program code that executes the technical solutions of the embodiments of this application, and its execution is controlled by the processor 111. The various types of computer program code being executed can also be considered as drivers for the processor 111. For example, the processor 111 executes the computer program code stored in the memory 112 to implement the technical solutions of the embodiments of this application.

[0205] Transceiver 113 can be used to support the reception or transmission of radio frequency (RF) signals between network elements and other devices. Transceiver 113 can be connected to antenna 115. Transceiver 113 includes a transmitter Tx and a receiver Rx. Specifically, one or more antennas 115 can receive RF signals. The receiver Rx of transceiver 113 is used to receive the RF signals from the antennas, convert the RF signals into digital baseband signals or digital intermediate frequency (IF) signals, and provide the digital baseband signals or IF signals to the processor 111 so that the processor 111 can perform further processing on the digital baseband signals or IF signals, such as demodulation and decoding. In addition, the transmitter Tx in transceiver 113 is also used to receive modulated digital baseband signals or IF signals from processor 111, convert the modulated digital baseband signals or IF signals into RF signals, and transmit the RF signals through one or more antennas 115. Specifically, the receiver Rx can selectively perform one or more stages of downmixing and analog-to-digital conversion on the radio frequency signal to obtain a digital baseband signal or a digital intermediate frequency (IF) signal. The order of the downmixing and IF conversion processes is adjustable. The transmitter Tx can selectively perform one or more stages of upmixing and digital-to-analog conversion on the modulated digital baseband signal or digital IF signal to obtain a radio frequency signal. The order of the upmixing and IF conversion processes is also adjustable. The digital baseband signal and the digital IF signal can be collectively referred to as digital signals.

[0206] Figure 8 illustrates an example of the composition of a terminal device provided in an embodiment of this application. This terminal device may be, for example, a mobile phone, a smart wearable device (such as a smartwatch), etc. Taking a mobile phone as an example, the terminal device may include a processor 310, an external memory interface 320, an internal memory 321, a display screen 330, a camera 340, antenna 1, antenna 2, a mobile communication module 350, and a wireless communication module 360, etc.

[0207] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device. In other embodiments, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0208] Processor 310 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, time-frequency codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0209] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the terminal device. In other embodiments of this application, the terminal device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0210] The external storage interface 320 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external storage card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, music, time and frequency files can be saved on the external storage card.

[0211] Internal memory 321 can be used to store executable program code, including instructions. Processor 310 executes various functional applications and data processing of the terminal device by running the instructions stored in internal memory 321. Internal memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the terminal device (such as time-frequency stream data), etc. Furthermore, internal memory 321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 310 executes various functions and data processing of the terminal device by running instructions stored in internal memory 321 and / or instructions stored in memory located within the processor.

[0212] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor, and baseband processor.

[0213] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0214] The mobile communication module 350 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on terminal devices. The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 350 may be housed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be housed in the same device.

[0215] In some embodiments, the terminal device initiates or receives call requests via the mobile communication module 350 and the antenna 1.

[0216] Furthermore, an operating system runs on top of the aforementioned components. Examples include iOS, Android, and Windows operating systems. Applications can be installed and run on this operating system. Those skilled in the art will understand that, for the sake of convenience and brevity, explanations and beneficial effects of the relevant content in any of the terminal devices provided above can be found in the corresponding method embodiments provided above, and will not be repeated here.

[0217] Furthermore, this application also provides a computer-readable storage medium storing instructions that, when executed on one or more computing devices, cause the one or more computing devices to perform the AI ​​model monitoring method described in the above embodiments.

[0218] Furthermore, this application also provides a computer program product. When executed by one or more computing devices, the computer program product enables the computing devices to execute any of the aforementioned AI model monitoring methods. This computer program product can be a software installation package. When any of the aforementioned AI model monitoring methods needs to be used, the computer program product can be downloaded and executed on a computer.

[0219] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0220] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0221] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0222] The system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

Claims

1. An AI model monitoring method, characterized in that, The method is applied to a first network device, including: The terminal device receives a first request message sent by the terminal device, the first request message being used to request monitoring of an AI model with positioning function, and the terminal device deploys the AI ​​model. A first test sample is determined from the test sample set based on the first request message. The location label corresponding to the first test sample is located within the effective area corresponding to the AI ​​model. The test sample set includes at least test samples determined based on samples sent by the terminal device, and the confidence level of the location label corresponding to the test sample is greater than the confidence level threshold. Send a first instruction to the terminal device, the first instruction being used to instruct the terminal device to perform positioning based on the AI ​​model and the first test sample, and obtain positioning results; The system receives a second request message sent by the terminal device, the second request message being used to request a determination of whether the positioning result output by the AI ​​model is reliable; The AI ​​model is judged based on the positioning error to determine whether it is reliable, and the judgment result is sent to the terminal device. The positioning error is determined based on the positioning result and the location label corresponding to the first test sample.

2. The method according to claim 1, characterized in that, The first instruction includes the sample identifier of the first test sample.

3. The method according to claim 1, characterized in that, The first instruction includes the sample identifier of the first test sample and the location label corresponding to the first test sample.

4. The method according to claim 1, characterized in that, The first instruction includes the first test sample and the location label corresponding to the first test sample.

5. The method according to any one of claims 1-4, characterized in that, The second request message includes the positioning error, which is determined by the terminal device based on the positioning result and the location tag corresponding to the first test sample.

6. The method according to any one of claims 1-4, characterized in that, The second request message includes the location result, and the method further includes: Based on the positioning results and the location label corresponding to the first test sample, the positioning error is determined.

7. The method according to claims 1-6, characterized in that, The step of obtaining a judgment result on whether the AI ​​model is reliable based on the positioning error includes: If the positioning error is greater than the error threshold, the distribution difference between the test sample set and the training sample set is determined, and the AI ​​model is generated based on the training sample set. If the distribution difference is less than the difference threshold, the factor causing the AI ​​model to be unreliable is determined as the first factor; Otherwise, the factor that causes the AI ​​model to be unreliable is determined as the second factor, and the judgment result includes the factor that causes the AI ​​model to be unreliable.

8. The method according to claim 7, characterized in that, The method further includes: If the first factor causes the AI ​​model to be unreliable, the AI ​​model shall be retrained based on the test sample set and the training sample set. If the second factor causes the AI ​​model to be unreliable, the AI ​​model is retrained based on the test sample set and the target training sample set. The target training sample set is a subset of the training sample set, and the regions indicated by the location labels of the training samples in the target training sample set do not overlap with the regions indicated by the location labels of the test samples in the test sample set. The retrained AI model is sent to the terminal device.

9. The method according to any one of claims 1-8, characterized in that, The step of obtaining a judgment result on whether the AI ​​model is reliable based on the positioning error includes: If the positioning error is less than the error threshold, determine whether the target test sample set is not empty. The target test sample set is a subset of the test sample set, and the area indicated by the location label corresponding to the test sample in the target test sample set does not intersect with the area indicated by the location label corresponding to the training sample in the training sample set. If the target test sample set is not empty, it is determined that the AI ​​model needs to be trained using the target test sample set, and the determination result includes an instruction to train the AI ​​model.

10. The method according to claim 9, characterized in that, The method further includes: The AI ​​model is trained using the target test sample set; The trained AI model is sent to the terminal device.

11. The method according to any one of claims 1-10, characterized in that, The first request message includes the identification information of the valid region corresponding to the AI ​​model, and the step of determining the first test sample from the test sample set based on the first request message includes: The first test sample located in the effective region is selected from the test sample set based on the location label of the test sample.

12. The method according to any one of claims 1-11, characterized in that, The step of determining the first test sample from the test sample set based on the first request message includes: If the test sample set contains a target test sample or the training sample set contains a target training sample, an intersection region is determined based on the position labels of the test samples in the test sample set and the position labels of the training samples in the training sample set. The position labels corresponding to the target test sample and the position labels corresponding to the target training sample do not belong to the intersection region. The AI ​​model is generated based on the training sample set. The test sample located in the intersection region is determined as the first test sample.

13. The method according to any one of claims 1-12, characterized in that, If the test sample set also includes a second test sample, the second test sample being a test sample located in the effective area determined by the first network device based on the second sample set sent by the second network device, the first instruction also includes the second test sample, and the first instruction is further used to instruct the terminal device to perform positioning based on the AI ​​model and the second test sample to obtain positioning results.

14. The method according to any one of claims 1-13, characterized in that, The method further includes: The terminal device receives a third request message, which requests the first network device to obtain test samples for monitoring the AI ​​model. Obtain the test sample set; A notification message is sent to the terminal device, the notification message being used to notify the terminal device that test samples for monitoring the AI ​​model have been obtained.

15. The method according to any one of claims 1-14, characterized in that, The acquisition of the test sample set includes: Send a second instruction to the terminal device, the second instruction being used to instruct the terminal device to collect first target data, the data type of the first target data including at least the data type required by the terminal device when using the AI ​​model for positioning; The terminal device receives a first sample set, which includes a first sample and a sample identifier corresponding to the first sample. The first sample set is determined by the terminal device based on the first target data. The first sample in the first sample set with a confidence level greater than a confidence threshold is determined as the test sample, whereby the confidence level indicates the credibility of the location label corresponding to the first sample.

16. The method according to claim 15, characterized in that, The first sample set also includes the location label corresponding to the first sample and the confidence level corresponding to the location label.

17. The method according to claim 15 or 16, characterized in that, If at least one first sample in the first sample set does not have a corresponding location label, the method further includes: The location label and confidence level of each first sample in the at least one first sample are determined based on the localization algorithm.

18. The method according to any one of claims 15-17, characterized in that, The first target data includes measurement data, the timestamp of the measurement data acquisition, and configuration information.

19. The method according to claim 18, characterized in that, The measurement data includes cellular measurement data, and the configuration information includes at least one of the following: collection start and end time and collection frequency.

20. The method according to claim 19, characterized in that, The cellular measurement data includes cell identifiers and reference signal measurement characteristics.

21. The method according to any one of claims 18-20, characterized in that, The measurement data also includes one or more of the following: Global Navigation Satellite System (GNSS) measurement data, Wi-Fi measurement data, Bluetooth measurement data, and sensor measurement data.

22. The method according to claim 21, characterized in that, The GNSS measurement data includes location labels and confidence levels; The Wi-Fi measurement data and the Bluetooth measurement data respectively include the wireless access point (AP) identifier and the received signal strength (RSSI); The sensor measurement data includes measurement data acquired based on at least one of the following sensors: accelerometer, gyroscope, and magnetometer.

23. The method according to any one of claims 15-22, characterized in that, The method further includes: Send a third instruction to the second network device, the third instruction being used to instruct the second network device to collect second target data, the data type of the second target data including at least the data type required for positioning using an AI model, and the location of the second network device being fixed; The system receives a second sample set sent by the second network device. The second sample set includes a second sample, a sample identifier corresponding to the second sample, and a location tag corresponding to the second sample. The location tag indicates the location of the second network device. Each second sample in the second sample set is selected as a test sample.

24. The method according to claim 23, characterized in that, The second target data includes cellular measurement data, location tags, and timestamps.

25. The method according to any one of claims 14-24, characterized in that, The third request message includes the identification information of the effective region corresponding to the AI ​​model, and the first sample refers to the sample located in the effective region.

26. The method according to claim 25, characterized in that, The identification information of the effective area corresponding to the AI ​​model includes the location information of the effective area and / or a list of cell identifiers within the effective area.

27. The method according to claim 25 or 26, characterized in that, The third instruction includes the identification information of the effective region corresponding to the AI ​​model, and the second sample refers to the sample located within the effective region.

28. The method according to any one of claims 24-27, characterized in that, If the terminal device does not agree to collect the first target data, the method further includes: The first network device receives a feedback message sent by the terminal device, the feedback message requesting the first network device to reconfigure the target data to be collected; The third target data to be collected is determined based on the feedback message. The third target data is different from the first target data. The data type of the third target data includes at least the data type required for localization using the AI ​​model. A fourth instruction is sent to the terminal device, the fourth instruction being used to instruct the terminal device to collect the third target data.

29. The method according to claim 27, characterized in that, The feedback message carries the reason for disagreeing with the collection of the first target data.

30. The method according to any one of claims 1-29, characterized in that, The first network device is a location management function (LMF) network element.

31. An AI model monitoring method, characterized in that, The method is applied to a terminal device and includes: A first request message is sent to a first network device, the first request message being used to request monitoring of an AI model with positioning function, and the terminal device deploys the AI ​​model; The terminal device receives a first instruction sent by the first network device, instructing it to perform localization using the AI ​​model. The first instruction instructs the terminal device to perform localization based on the AI ​​model and a first test sample. The location tag corresponding to the first test sample is located within the effective area corresponding to the AI ​​model. The first test sample is determined by the first network device from a set of test samples based on the first request message. The set of test samples includes at least test samples determined based on samples sent by the terminal device, and the confidence level of the test samples is greater than a confidence level threshold. Based on the AI ​​model and the first test sample, the localization result is obtained. Send a second request message to the first network device, the second request message being used to request a determination of whether the positioning result output by the AI ​​model is reliable; The system receives a judgment result from the first network device regarding the reliability of the AI ​​model. The judgment result is determined by the first network device based on a positioning error, which is determined based on the positioning result and the location label corresponding to the first test sample.

32. The method according to claim 31, characterized in that, The first instruction includes the sample identifier of the first test sample.

33. The method according to claim 31, characterized in that, The first instruction includes the sample identifier of the first test sample and the location label corresponding to the first test sample.

34. The method according to claim 31, characterized in that, The first instruction includes the first test sample and the location label corresponding to the first test sample.

35. The method according to any one of claims 31-34, characterized in that, The method further includes: The positioning error is determined based on the positioning result and the location label corresponding to the first test sample, and the second request message includes the positioning error.

36. The method according to any one of claims 31-35, characterized in that, The second request message includes the positioning result, and the positioning error is determined by the first network device based on the positioning result and the location label corresponding to the first test sample.

37. The method according to any one of claims 31-36, characterized in that, If the judgment result indicates that the AI ​​model is untrustworthy, the judgment result includes factors that cause the AI ​​model to be untrustworthy; If the judgment result indicates that the AI ​​model is trustworthy, the judgment result also includes an indication to train the AI ​​model.

38. The method according to claim 37, characterized in that, The method further includes: Receive the retrained AI model sent by the first network device.

39. The method according to claim 37, characterized in that, The method further includes: If the first factor causing the AI ​​model to be unreliable is to obtain the test sample set and the training sample set, and retrain the AI ​​model based on the test sample set and the training sample set; If the unreliability of the AI ​​model is a second factor, the test sample set and the target training sample set are obtained, and the AI ​​model is retrained based on the test sample set and the target training sample set. The target training sample set is a subset of the training sample set, and the regions indicated by the location labels of the training samples in the target training sample set do not intersect with the regions indicated by the location labels of the test samples in the test sample set.

40. The method according to claim 37, characterized in that, The method further includes: If the judgment result indicates that the AI ​​model is trustworthy, a target test sample set is obtained, and the AI ​​model is trained using the target test sample set. The target test sample set is a subset of the test sample set, and the regions indicated by the location labels corresponding to the test samples in the target test sample set do not intersect with the regions indicated by the location labels corresponding to the training samples in the training sample set.

41. The method according to any one of claims 31-40, characterized in that, The first request message includes the identification information of the effective region corresponding to the AI ​​model, and the first request message is used to request the use of test samples within the effective region to monitor the AI ​​model.

42. The method according to any one of claims 31-41, characterized in that, If the test sample set also includes a second test sample, the first instruction also includes the second test sample, which is determined by the first network device based on the second sample set sent by the second network device.

43. The method according to any one of claims 31-42, characterized in that, The method further includes: Send a third request message to the first network device, the third request message being used to request the first network device to obtain test samples for monitoring the AI ​​model; The system receives a notification message sent by the first network device, the notification message being used to notify that test samples for monitoring the AI ​​model have been obtained.

44. The method according to claim 43, characterized in that, The method further includes: The terminal device receives a second instruction sent by the first network device, the second instruction being used to instruct the terminal device to collect first target data, the data type of the first target data including at least the data type required by the terminal device when using the AI ​​model for positioning; The first target data is acquired, and a first sample set is determined based on the first target data. The first sample set includes a first sample and a sample identifier corresponding to the first sample. Send the first sample set to the first network device.

45. The method according to claim 44, characterized in that, The first sample set also includes the location label corresponding to the first sample and the confidence level corresponding to the location label.

46. ​​The method according to claim 44 or 45, characterized in that, The first target data includes measurement data, the timestamp of the measurement data acquisition, and configuration information.

47. The method according to claim 46, characterized in that, The measurement data includes cellular measurement data, and the configuration information includes at least one of the following: collection start and end time and collection frequency.

48. The method according to claim 47, characterized in that, The cellular measurement data includes cell identifiers and reference signal measurement characteristics.

49. The method according to claim 47 or 48, characterized in that, The measurement data also includes one or more of the following: Global Navigation Satellite System (GNSS) measurement data, Wi-Fi measurement data, Bluetooth measurement data, and sensor measurement data.

50. The method according to claim 49, characterized in that, The GNSS measurement data includes location labels and confidence levels; The Wi-Fi measurement data and the Bluetooth measurement data respectively include the wireless access point (AP) identifier and the received signal strength (RSSI); The sensor measurement data includes measurement data acquired based on at least one of the following sensors: accelerometer, gyroscope, and magnetometer.

51. The method according to any one of claims 44-50, characterized in that, The method further includes: If the terminal device does not agree to collect the first target data, it sends a feedback message to the first network device, and the feedback message requests the first network device to reconfigure the target data to be collected. The terminal device receives a fourth instruction sent by the first network device, the fourth instruction being used to instruct the terminal device to collect third target data.

52. The method according to claim 51, characterized in that, The feedback message carries the reason for disagreeing with the collection of the first target data.

53. An AI model monitoring method, characterized in that, The method is applied to a second network device, including: The system receives a third instruction sent by a first network device, the third instruction being used to instruct the second network device to collect second target data, the data type of the second target data including at least the data type required for positioning using an AI model, and the location of the second network device being fixed. Acquire the second target data, and determine the second sample set based on the second target data; The second sample set is sent to the first network device so that the first network device determines test samples based on the second sample set and uses the test samples to monitor the AI ​​model. The second sample set includes a second sample, a sample identifier corresponding to the second sample, and a location tag corresponding to the second sample. The location tag indicates the location of the second network device.

54. The method according to claim 53, characterized in that, The second target data includes cellular measurement data, location tags, and timestamps.

55. The method according to claim 53 or 54, characterized in that, The second instruction includes the identification information of the effective region corresponding to the AI ​​model, and the second sample refers to the sample located within the effective region.

56. A network device, characterized in that, include: A transceiver for performing the receiving and transmitting operations in the method of any one of claims 1-30; A processor for performing operations other than the receiving operation and the sending operation in the method of any one of claims 1-30; or, A transceiver for performing the receiving and transmitting operations in the method of any one of claims 53-55; A processor for performing operations other than the receiving operation and the sending operation in the method of any one of claims 53-55; or.

57. A terminal device, characterized in that, include: A transceiver for performing the receiving and transmitting operations in any one of claims 31-52; A processor for performing operations other than the receiving operation and the sending operation in the method of any one of claims 31-52.

58. A communication system, characterized in that, This includes terminal equipment and network equipment; The network device is configured to perform the method according to any one of claims 1-30, or to perform the method according to any one of claims 53-55; The terminal device is used to perform the method according to any one of claims 31-52.

Images