Ai / ML verification method and apparatus in wireless communication system

Abstract

Disclosed is an AI / ML verification method carried out by a first communication node including a communication unit and a processor. The disclosed method comprises the steps of: generating AI / ML configuration information including a first threshold value representing an AI / ML performance criterion applied to normal operation of AI / ML and a second threshold value representing an AI / ML performance criterion applied to AI / ML verification, the second threshold value being distinct from the first threshold value; and transmitting the AI / ML configuration information, wherein the second threshold value includes a threshold value for determining a condition under which at least one of reversion from AI / ML verification to non-AI / ML, deactivation of AI / ML, change of an AI / ML model, and AI / ML-related reporting occurs.

Description

AI / ML verification method and device in a wireless communication system

[0001] The present disclosure relates to a method for verifying AI / ML in a wireless communication system, and more specifically, to an AI / ML management method capable of verifying / testing the functions, performance, etc. of AI / ML in a wireless communication system that supports AI / ML.

[0002] A communication system may include a core network, base stations (e.g., macro base stations, small base stations, relays, etc.), terminals, etc. Communication between a base station and a terminal may be performed based on various radio access technologies (RATs) (e.g., 4G communication technology, 5G communication technology, 6G communication technology, WLAN (wireless local area network) technology, WPAN (wireless personal area network) technology, etc.).

[0003] 3GPP selected AI / ML as a Study Item for 5G Release 18 and conducted research and discussions. AI / ML technology utilizes artificial intelligence and machine learning to streamline network operations and optimize service quality. 3GPP is prioritizing the introduction of AI / ML in scenarios such as 5G beam management, positioning, channel state prediction, and channel state compression, expecting improvements in system throughput, resource utilization, and coverage.

[0004] In addition, 3GPP has begun full-scale 6G research and is considering AI / ML as the core of the AI-native architecture for 6G. Through this, based on AI / ML frameworks developed in 5G, it is researching data management, model deployment, and Air Interface optimization (e.g., CSI compression, mobility enhancement), and aims to enable autonomous AI operations, maximize network efficiency, and support new use cases in 6G systems.

[0005] Standardization organizations such as 3GPP and the O-RAN Alliance are defining a Life Cycle Management (LCM) framework for AI / ML functions to apply AI / ML to various network functions and to operate them efficiently.

[0006] LCM includes a series of management procedures such as deployment, enabling, performance monitoring, disabling, and removal of AI / ML models. LCM also includes features to replace the model with existing logic-based functions, switch to another AI / ML model with better performance, or fallback to a legacy system that does not utilize AI / ML if the model's performance falls below a predefined threshold.

[0007] AI / ML models can be modified or updated even after deployment. Additionally, new AI / ML models may be deployed. There is no guarantee that the performance of the device will improve during use even after deployment or updates. Therefore, it is essential to perform verification and testing of the AI / ML when applying new or updated AI / ML models to communication nodes.

[0008] According to at least one embodiment, a method and apparatus for verifying AI / ML are disclosed.

[0009] An AI / ML verification method is disclosed, which is performed by a first communication node including a communication unit and a processor.

[0010] The above method includes the step of generating AI / ML configuration information including a first threshold value representing an AI / ML performance criterion applied to the normal operation of AI / ML and a second threshold value representing an AI / ML performance criterion applied to the verification of a distinguished AI / ML, and transmitting said AI / ML configuration information, wherein the second threshold value may include a threshold value for determining a condition in which at least one of a return to non-AI / ML, deactivation of AI / ML, change of AI / ML model, and AI / ML-related reporting occurs during the verification of AI / ML.

[0011] The second threshold value may be set to represent a relatively lower AI / ML performance standard compared to the first threshold value.

[0012] The second threshold value may be set such that, compared to the first threshold value, the range in which conditions are satisfied in which at least one of the following occurs during AI / ML verification is a return to non-AI / ML, the deactivation of AI / ML, a change in the AI / ML model, and an AI / ML-related report is satisfied is reduced.

[0013] The above AI / ML configuration information may include instruction information for a first metric representing AI / ML performance during normal operation of AI / ML, and instruction information for a second metric that is set separately from the first metric and represents AI / ML performance during verification of AI / ML.

[0014] The above second metric may have a relatively higher computational complexity compared to the above first metric.

[0015] The above second metric can be expressed as a relatively large amount of data compared to the above first metric.

[0016] The first metric includes at least one of a confidence score, output deviation, and number of trigger occurrences for an AI / ML model, and the second metric may include at least one of an inference accuracy, latency, and loss function value of an AI / ML model.

[0017] The above AI / ML configuration information may include at least one of information representing the signaling logic for AI / ML-related reporting during normal operation of AI / ML and information representing the signaling logic for AI / ML-related reporting during verification of AI / ML.

[0018] In the normal operation of the above AI / ML, the signaling logic for AI / ML-related reporting and the signaling logic for AI / ML-related reporting in the verification of the above AI / ML can each indicate a comparison direction used to trigger AI / ML-related reporting.

[0019] The method includes the steps of: transmitting a terminal function request message to a second communication node; obtaining terminal function information from the second communication node; transmitting a first configuration information to the second communication node, which includes AI / ML function information allowed by the first communication node based on the terminal function information; and obtaining an applicable AI / ML function report from the second communication node, wherein the AI / ML configuration information may be included in the first configuration information.

[0020] The above method comprises the steps of: transmitting a terminal function request message to a second communication node; obtaining terminal function information from the second communication node; transmitting a first configuration information to the second communication node including AI / ML function information allowed by the first communication node based on the terminal function information; obtaining an applicable AI / ML function report from the second communication node; and transmitting a second configuration information including an inference configuration based on the applicable AI / ML function report, wherein the AI / ML configuration information may be included in the second configuration information.

[0021] In another aspect, an AI / ML verification method is disclosed that is performed by a second communication node including a communication unit and a processor.

[0022] The disclosed method comprises the steps of: obtaining AI / ML configuration information from a first communication node, the first threshold representing an AI / ML performance criterion applied to the normal operation of AI / ML, and the second threshold representing an AI / ML performance criterion applied to the verification of AI / ML. The method further comprises the step of performing AI / ML verification based on the AI / ML configuration information, wherein the second threshold includes a threshold for determining a condition in which at least one of a return to non-AI / ML, deactivation of AI / ML, change of AI / ML model, and AI / ML-related reporting occurs during the verification of AI / ML.

[0023] The second threshold value may be set to represent a relatively lower AI / ML performance standard compared to the first threshold value.

[0024] The second threshold value can be set to represent a relatively higher AI / ML performance standard compared to the first threshold value.

[0025] The above AI / ML configuration information may include instruction information for a first metric representing AI / ML performance during normal operation of AI / ML, and instruction information for a second metric that is set separately from the first metric and represents AI / ML performance during verification of AI / ML.

[0026] The above AI / ML configuration information may include at least one of information representing the signaling logic for AI / ML-related reporting during normal operation of AI / ML and information representing the signaling logic for AI / ML-related reporting during verification of AI / ML.

[0027] The above method may include the step of performing an update of an AI / ML model; the step of generating an AI / ML update report based on the update information of the AI / ML model; and the step of transmitting the AI / ML update report.

[0028] The above AI / ML update report may include at least one of identification information of the updated AI / ML model, information on the time when the AI / ML model was updated, and information on the cause of the update of the AI / ML model.

[0029] The above AI / ML update report may include information about when the updated AI / ML model is applied by the second communication node.

[0030] The identification information of the above-mentioned updated AI / ML model may be indicated by at least one of an AI / ML model ID, an AI / ML function ID, and an association ID.

[0031] According to at least one embodiment, the threshold value used for AI / ML verification and the threshold value used for the normal operation of AI / ML can be set differently. This prevents unnecessary return operations during the AI / ML verification process and allows for the acquisition of sufficient data. Additionally, signaling overhead can be reduced during the AI / ML verification process.

[0032] According to at least one embodiment, distinct metrics may be used in the AI / ML validation environment and the AI / ML normal operation environment, respectively. By using distinct metrics, signaling overhead and metric computation resources can be reduced in the AI / ML normal operation. In AI / ML validation, the actual effects of model updates or fine-tuning can be accurately evaluated.

[0033] According to at least one embodiment, AI / ML configuration information may include a field representing signaling logic for AI / ML-related reporting. Flexibility in communication network operation may be increased by using a parameter representing a comparison direction used to trigger AI / ML-related reporting.

[0034] According to at least one embodiment, the second communication node can explicitly signal an update report after updating the AI / ML model. By signaling the update report, the latest information regarding the AI / ML model is shared between the first and second communication nodes, thereby facilitating AI / ML verification and LCM full-cycle management.

[0035] FIG. 1 is a drawing showing a wireless communication system (100) according to an exemplary embodiment.

[0036] FIG. 2 is a block diagram illustrating the configuration of a communication node (200) that constitutes a communication system in an exemplary manner.

[0037] Figure 3 is a diagram illustrating an exemplary 5G network architecture.

[0038] Figure 4 is a diagram showing a 6G network architecture.

[0039] Figure 5 is a block diagram showing the full cycle of AI / ML functions in a mobile communication system.

[0040] Figure 6 is a diagram showing the prediction error according to the training of an AI / ML model.

[0041] FIG. 7 is a flowchart illustrating an AI / ML verification method performed by a communication node according to an exemplary embodiment.

[0042] Figure 8 is a diagram illustrating an exemplary configuration method of IE included in AI / ML configuration information.

[0043] Figure 9 is a diagram showing an example of the implementation of AI / ML configuration information.

[0044] Figure 10 is a diagram illustrating an exemplary configuration method of IE included in AI / ML configuration information.

[0045] Figure 11 is a diagram showing an example of the implementation of AI / ML configuration information.

[0046] Figure 12 is a flowchart illustrating an AI / ML verification method that includes a process of transmitting a signaling indicating an AI / ML model update.

[0047] An AI / ML verification method is disclosed, which is performed by a first communication node including a communication unit and a processor.

[0048] The above method includes the step of generating AI / ML configuration information including a first threshold value representing an AI / ML performance criterion applied to the normal operation of AI / ML and a second threshold value representing an AI / ML performance criterion applied to the verification of a distinguished AI / ML, and transmitting said AI / ML configuration information, wherein the second threshold value may include a threshold value for determining a condition in which at least one of a return to non-AI / ML, deactivation of AI / ML, change of AI / ML model, and AI / ML-related reporting occurs during the verification of AI / ML.

[0049] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0050] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0051] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0052] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0053] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0054] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate an overall understanding of the present invention, the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0055] Table 1 shows the abbreviations used in the present disclosure.

[0056] 약어전체 이름AI / MLArtificial Intelligence / Machine Learning3GPP3rd Generation Partnership ProjectACKAcknowledgementAFApplication FunctionAIArtificial IntelligenceAMFAccess and Mobility Management FunctionAUSFAuthentication Server FunctionBWPBandwidth PartC-RNTICell RNTICSIChannel State InformationCSI-RSChannel State Information Reference SignalCLICross link InterferenceCEControl ElementDCIDownlink Control InformationIEInformation elementMACMedium Access ControlMLMachine LearningNSSFNetwork Slicing Selection FunctionNEFNetwork Exposure FunctionNRFNF Repository FunctionPCFPolicy Control FunctionPDCCHPhysical Downlink Control ChannelPDSCHPhysical Downlink Shared ChannelPDUProtocol Data UnitPRACHPhysical Random Access ChannelPT-RSPhase Tracking Reference SignalPUCCHPhysical Uplink Control ChannelPUSCHPhysical Uplink Shared ChannelRARandom AccessRACHRandom Access ChannelRANRadio Access NetworkRBResource BlockRRCRadio Resource ControlRSRPReference Signal Received PowerRSRQReference Signal ReceivedQualityRSSIReceived Signal Strength IndicatorSBFDSub-Band Full DuplexSISelf InterferenceSIBSystem information blockUEUser EquipmentDLDownlinkULUplinkRARRandom Access ResponseRRMRadio Resource ManagementRRCRadio Resource ControlRSRPReference Signal Received PowerQoSQuality of ServiceDRBData Radio BearerDAPSDual Active Protocol StackTDDTime Division DuplexingFDDFrequency Division DuplexingSSBSS BlockCQIChannel Quality IndicatorPMIPrecoding Matrix IndicatorLILayer IndicatorLCMLife Cycle ManagementRIRank IndicatorQCLQuasi coLocation

[0057] In the following, "validation of AI / ML" refers to a procedure for evaluating or testing the performance of AI / ML models identified by at least one of a specified AI / ML model ID, an AI / ML function ID, and an associated ID.

[0058] The verification of AI / ML may include verification procedures conducted in a designated test environment simulating a commercial communication network and verification procedures conducted on a commercial communication network. The verification of AI / ML may include at least one of a conformance test performed in advance before the AI / ML model is applied to the field and a post-deployment test performed after the AI / ML model is applied to the field.

[0059] FIG. 1 is a drawing showing a wireless communication system (100) according to an exemplary embodiment.

[0060] Referring to FIG. 1, a wireless communication system (100) may be composed of a plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4). Here, a communication node refers to a node capable of transmitting and receiving signals in the wireless communication system (100), and each of the plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) may support at least one communication protocol. For example, each of the multiple communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) can support cellular communication (e.g., LTE (long term evolution), LTE-A (advanced), 5G NR, 5G-Advanced, etc. as defined in the 3GPP (3rd generation partnership project) standard).

[0061] For example, each of the multiple communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) can support a CDMA (code division multiple access) based communication protocol, a WCDMA (wideband CDMA) based communication protocol, a TDMA (time division multiple access) based communication protocol, a FDMA (frequency division multiple access) based communication protocol, an OFDM (orthogonal frequency division multiplexing) based communication protocol, an OFDMA (orthogonal frequency division multiple access) based communication protocol, a SC (single carrier)-FDMA based communication protocol, a NOMA (non-orthogonal multiple access) based communication protocol, a SDMA (space division multiple access) based communication protocol, a SBFD (sub-band full duplex), AI / ML, etc.

[0062] A plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) may include a plurality of base stations (110-1, 110-2, 110-3) and a plurality of terminals (120-1, 120-2, 120-3, 120-4). Each of the base stations (110-1, 110-2, 110-3) may form a cell. The cell may include a small cell, a macro cell, a pico cell, a femto cell, etc., but the embodiments are not limited thereto. For example, the first terminal (120-1) and the second terminal (120-2) may be within the coverage of the first base station (110-1), the third terminal (120-3) may be within the coverage of the second base station (110-2), and the fourth terminal (120-4) may be within the coverage of the third base station (110-3). Additionally, some of the multiple terminals (120-1, 120-2, 120-3, 120-4) may support AI / ML functions, while others may not support AI / ML functions. 3GPP is conducting technology research in 6G, including new interfaces, advanced sensing, AI / ML-based automation, and sustainable connectivity.

[0063] Multiple base stations (110-1, 110-2, 110-3) may be referred to as gNodeB (gNB), NodeB, evolved NodeB, BTS (base transceiver station), radio base station, radio transceiver, access point, access node, roadside unit (RSU), RRH (radio remote head), TP (transmission point), TRP (transmission and reception point), relay node, etc. Multiple base stations (110-1, 110-2, 110-3) may form a Radio Access Network (RAN). The Radio Access Network may be connected to a core network.

[0064] Each of the multiple terminals (120-1, 120-2, 120-3, 120-4) may be referred to as a terminal, access terminal, mobile terminal, station, subscriber station, mobile station, portable subscriber station, node, device, etc.

[0065] The communication protocol support ranges of each of the multiple base stations (110-1, 110-2, 110-3) may differ from one another. For example, some of the multiple base stations (110-1, 110-2, 110-3) may support AI / ML functions while others do not. Similarly, the communication protocol support ranges of each of the terminals (120-1, 120-2, 120-3, 120-4) may also differ from one another. For example, some of the multiple terminals (120-1, 120-2, 120-3, 120-4) may support AI / ML functions while others do not.

[0066] FIG. 2 is a block diagram illustrating the configuration of a communication node (200) that constitutes a communication system. At least some of the communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3) shown in FIG. 1 may correspond to the communication node (200) shown in FIG. 2.

[0067] Referring to FIG. 2, the communication node (200) may include at least one processor (210), a memory (220), and a transceiver (230) that is connected to a network to perform communication. Additionally, the communication node (200) may further include an input interface device (240), an output interface device (250), a storage device (260), etc. Each component included in the communication node (200) may be connected by a bus (270) to communicate with one another.

[0068] The processor (210) can execute a program command stored in at least one of the memory (220) and the storage device (260). The processor (210) may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Each of the memory (220) and the storage device (260) may be composed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (220) may be composed of at least one of read-only memory (ROM) and random access memory (RAM).

[0069] FIG. 3 is a diagram illustrating an exemplary 5G network architecture. Referring to FIG. 3, the 5G network architecture may include an NSSF providing a network slicing control solution, an NEF providing a network function opening solution, an NRF providing an NF interoperability control solution within the 5G network, a PCF providing a data packet flow policy control solution, a UDM providing a user information and policy management solution, an AMF providing a network access and mobility management solution, an SMF providing a terminal / network session management solution, a UPF providing a solution for user packet routing and terminal connectivity between base stations, a radio access network (RAN), and a user terminal (UE).

[0070] N1 reference point is defined to transmit signaling between the UE and the AMF, N2 is defined as the reference point to connect the RAN access node and the AMF, and N3 is defined as the reference point to connect the RAN access node and the UPF.

[0071] 3GPP is currently conducting 6G research, and the 6G network architecture plans to integrate AI-native design, advanced sensing, and ultra-low latency communication based on the existing 5G structure. In 6G, the N4 interface between the User Plane Function (UPF) and Session Management Function (SMF) and the N3 interface between the RAN and UPF will be optimized, and AI / ML-based dynamic resource management and network slicing will be enhanced. Furthermore, 6G is highly likely to support industrial use cases (e.g., autonomous driving, holographic communication) by introducing new Network Functions (NFs) and interfaces.

[0072] Figure 4 is a diagram showing a 6G network architecture.

[0073] Referring to FIG. 4, the 6G network architecture provides AI-based end-to-end network management and automation services and analysis information exposure, and provides support functions based on cloud infrastructure, transmission network, data pipeline, and common platform functions. The 6G network architecture includes a 6G terminal (610), a 6G radio access network (620, RAN), and an AI native cloud-based 6G core network (630, CN), and can also be interconnected with an existing 5G network as needed.

[0074] Here, the opening of AI-based service and analysis information supports the improvement of network efficiency and service quality by providing AI-processed network status information and analysis results to service providers or external systems. In addition, 6G core networks operating in a cloud-native environment enable dynamic network resource management, autonomous service orchestration, and efficient user service quality assurance through network functions with embedded artificial intelligence (AI).

[0075] This architecture strengthens the connection with the data network to provide flexibility in data flow management and defines an efficient communication path and management interface between the 6G terminal (610) and the data network. In particular, the AI-based E2E end-to-end management function enables autonomous and intelligent network control across the entire network area (terminal-RAN-CN-data network).

[0076] FIG. 5 is a block diagram illustrating the full lifecycle of AI / ML functions in a mobile communication system. Operations such as updating, changing, managing, verifying, and reporting of AI / ML-related models described above can all be performed within the full lifecycle management (LCM) framework shown in FIG. 5. The AI / ML function blocks shown in FIG. 5 can be implemented by at least one of the communication nodes of the mobile communication system. For example, the AI / ML function blocks shown in FIG. 5 can be implemented by at least one of a base station and a terminal.

[0077] When the first terminal (120-1) is connected to the service area of ​​the first base station (110-1) that supports AI / ML functions, the first terminal (120-1) or the first base station (110-1) can activate at least some of the function blocks (410, 412, 414, 416, 418) of FIG. 5.

[0078] The terminal (120-1) can collect data related to the wireless channel by activating the data collection block (410). The terminal (120-1) can collect data related to wireless channel information from reference signals (CSI-RS, SSB, PT-RS, PRS, downlink reference signals, etc.). When the base station (110-1) activates the data collection block (410), it can collect data related to wireless channel information from reference signals (SRS, PRS, PT-RS) and channel status reports, etc.

[0079] Data collected from the data collection block (410) can be used as data for training / monitoring / inference. The model training block (412) can train an AI / ML model using the data collected from the data collection block (410). The model storage block (418) can store the trained AI / ML model. The stored model can be used by the inference block (416). The inference block (416) can infer information necessary for the operation of a wireless communication system based on the data collected from the data collection block (410).

[0080] The inference block (416) can report the inference results to the AI / ML management block (414). The AI / ML management block (414) can perform overall functions related to training the AI / ML model, selecting the AI / ML model, enabling / disabling the AI / ML model, and controlling other function blocks. The AI / ML management block (414) can monitor the performance of the AI / ML model to maintain the performance of the AI / ML and provide feedback on the training results to the model training block (412). If the performance of the monitored AI / ML model is low and retraining is required, the AI / ML management block (414) can request retraining from the model training block (412). The AI / ML management block (414) can determine the enabling / disabling of the AI / ML model and the selection / change of the model. The AI / ML management block (414) performs management instructions such as a fallback to an existing communication method that does not use AI / ML functions.

[0081] AI / ML models can be classified in various ways. For example, a model ID can be assigned to each AI / ML model. As another example, AI / ML models can be classified by function, and a functionality ID can be assigned to each function. Associated IDs may also be used to distinguish AI / ML models.

[0082] An Associated ID may represent a configuration for data collection to train an AI / ML model. An Associated ID can be used to ensure the identity between the training environment and the inference environment of the AI / ML model. A terminal (120-1) can verify the configuration for data collection from the Associated ID and collect data according to that configuration. The terminal (120-1) can train or develop an AI / ML model using the collected data based on the Associated ID.

[0083] AI / ML models possess the characteristics of overfitting and generalization. Overfitting refers to a situation where a model performs excellently only on a specific dataset but degrades on others. Conversely, generalization refers to the characteristic where performance can be guaranteed across various datasets. However, there are limitations to the generalization of AI / ML models. In other words, the performance of an AI / ML model can be guaranteed for data collected under a somewhat limited data collection configuration. Therefore, wireless communication networks can define a data collection configuration that guarantees the performance of AI / ML models and assign an Associated ID to each data collection configuration.

[0084] A wireless communication network can define a set (range) of generalizable datasets where the performance of AI / ML models is guaranteed, and determine a data collection configuration to correspond to this. The wireless communication network can define various data collection configurations and assign an Associated ID to each of them. The wireless communication network can train or develop AI / ML models according to the data collection configuration corresponding to each Associated ID.

[0085] The method for assigning Associated IDs may vary depending on the use case of the AI / ML model. For example, for AI / ML models used for beam management, data collection configurations may be determined based on the same downlink spatial domain transmission filter, reference signals in Quasi Co-location (QCL) status, or antenna panels used for transmission and reception. A different Associated ID may be assigned to each of the data collection configurations. In the example of CSI compression and prediction, data collection configurations may be determined based on SNR, CQI, PMI, RI representing channel environment status, and antenna panels used for transmission and reception. Associated IDs are uniquely defined within the network and can be classified into global Associated IDs representing the same environment and local Associated IDs valid only for certain cell groups / cells / sites / regions or equipment vendors. In the following, the Associated ID may correspond to at least one of the global Associated ID and the local Associated ID.

[0086] Each AI / ML model can be assigned a Model ID. An Associated ID can correspond to at least one Model ID. As another example, an Associated ID can correspond to multiple Model IDs. The Associated ID and Model ID can have a many-to-many correspondence.

[0087] For example, if the downlink spatial domain transmission filters are configured as a number (N), the Associated IDs for beam management can also be configured as a number (N), and the AI / ML models used for the corresponding AI / ML beam management can be configured as N corresponding. Similarly, if the base station (110-1) configures a number (N) of reference signal / resource sets that are not in a QCL state, the AI / ML models used for the corresponding AI / ML beam management and the associated IDs can be configured as N corresponding. On the other hand, if the AI / ML models have excellent generalization features and their performance is guaranteed across all datasets, they can be configured and operated with a small number (less than N) of AI / ML models.

[0088] In a 6G Massive MIMO environment, there is a Spatial Non-Stationarity (SNS) characteristic where channel statistics vary for each position or segment within the array. Therefore, in 6G networks, there is similarity in the wireless environment at the segment (= physical subarray or virtual division) level, and Associated IDs can be constructed based on this similarity. When the segment is single (whole array), there is no virtual division, and the time and frequency fields are fixed as the basic measurement units, the 6G Massive MIMO environment has a concept equivalent to the existing Associated ID.

[0089] A 6G Associated ID is defined by a combination of a core spatial context, an SNS segment context, a time / frequency window, and aggregation rules. Sharing the same 6G Associated ID implies that the data was processed under the same segment splitting, label, weighting, aggregator, and time / frequency window conditions. The core spatial context includes the same downlink spatial domain transmission filter, reference signal for QCL status, and transmit / receive antenna panels (including beamforming chains / modes if necessary), ensuring compatibility with 5G. The Spatial Non-Stationary Segment Context includes the definition and version of the segment splitting, the set of segment identifiers actually included, the version of the segment environment label generation algorithm, and weighting rules based on distance, path power, SNR, etc., and can add near / far distinctions to the concept as needed.

[0090] The 6G Associated ID is also uniquely defined within the network and, as before, can be classified into a global associated ID and a local associated ID limited to specific cell groups, cells, sites, regions, or network manufacturers. The relationship with AI / ML models is also extended, allowing for a 1:1 or N:N (many-to-many) correspondence between the 6G Associated ID and the model ID.

[0091] For example, in the case of beam management, if multiple downlink spatial domain transmission filters are configured, the array is divided into K segments, and a P90 aggregator is applied, the data collection configuration is identified in units of a fixed 6GAssociated ID that includes the filter, QCL, and panel, as well as the segment division and segment set, label algorithm, weighting rule, time / frequency window, and aggregator. In the case of CSI compression and prediction, since the segment label and weighting / aggregation rule affect the results in addition to the SNR, CQI, PMI, RI, and transmit / receive antenna panel, a 6GAssociated ID containing this context can function as an identification key for the data collection configuration.

[0092] In addition, when there is only one segment, no virtual partitioning, the aggregator is the Identity, and the time / frequency window is fixed as the basic measurement unit, the 6G Associated ID is reduced to be virtually identical to the existing Associated ID, so it is naturally backward compatible with 5G-centric operations. In short, the 6GAID can be described as an extended identifier that strictly guarantees reproducibility and equivalence judgment throughout the entire process of data collection, learning, inference, and verification by preserving the meaning of the associated identification used in 5G while adding the segment, window, and aggregation contexts necessary for SNS and PDV operations.

[0093] The operation of AI / ML models can be broadly classified into single-sided operation (one-side AI / ML model) and joint operation (two-side AI / ML model). In the case of a one-side AI / ML model, the AI / ML model is operated on only one side, either the base station or the terminal, whereas in the case of a two-side AI / ML model, the AI / ML model can be operated simultaneously on both the base station and the terminal.

[0094] Lifecycle Management (LCM) for AI / ML can be performed in various ways. First, the scope of LCM application can be determined by the AI / ML model ID. A base station or terminal can determine the scope of LCM application based on at least one AI / ML model ID. A base station or terminal can determine at least one AI / ML model to which LCM is applied based on at least one AI / ML model ID.

[0095] Secondly, the scope of application of AI / ML can be determined by a function ID or function identifier. Function-based LCM can be applied to AI / ML model(s) associated with at least one function without the need to precisely specify the AI / ML model between the base station and the terminal. The base station or the terminal can perform LCM-based operations for the AI / ML model(s) associated with a specific function.

[0096] Thirdly, the scope of application of AI / ML can be determined by an Associated ID. An Associated ID-based LCM can be applied to AI / ML model(s) corresponding to at least one Associated ID without the need to precisely specify the AI / ML model between the base station and the terminal. The base station or the terminal can perform LCM-based operations for the AI / ML model(s) corresponding to a specific Associated ID.

[0097]

[0098] Figure 6 is a diagram showing the prediction error according to the training of an AI / ML model.

[0099] In Figure 6, the horizontal axis represents the number of training cycles or epochs, and the vertical axis represents the prediction error of the AI / ML model. The L1 curve in Figure 6 represents the validation error of the AI / ML model, and the L2 curve represents the training error. Here, the training error represents the error between the value predicted by the AI / ML for the data used for training and the correct answer, and the validation error represents the error between the value predicted by the AI / ML for the validation data not used for training and the correct answer.

[0100] Referring to Figure 6, as the training cycle of an AI / ML model increases, the learning error tends to decrease, but the validation error tends to increase after a certain point. This phenomenon is called overfitting. In other words, if there are insufficient training cycles, underfitting occurs, and if there are too many training cycles, overfitting may occur.

[0101] In mobile communication environments, training data is mostly a partial dataset of environmental data acquired by communication nodes. If an AI / ML model is overfitted or underfitted to this training data, the reliability of the AI / ML model's inference results may decrease and performance may degrade.

[0102] To prevent such performance degradation, verification of AI / ML is necessary. During the verification process, monitoring and testing of AI / ML performance may be performed. Verification may be conducted in an actual wireless communication environment or in a test environment that simulates an actual wireless communication environment.

[0103] AI / ML validation may include post-deployment verification. Communication nodes may perform updates and / or fine-tuning of AI / ML models while operating AI / ML. The data used in the AI / ML training environment or conformance validation may differ from the data collected in the actual field (real-world communication environment). This may lead to data drift or data mismatch phenomena.

[0104] The communication node can perform fine-tuning or updates to the AI / ML model after the AI / ML model has been deployed. As another example, the communication node may change and / or switch the AI / ML model to a different AI / ML model than the existing one. In the cases described above, the communication node may perform a verification procedure after deployment.

[0105] The verification of AI / ML can be performed by a first communication node and a second communication node. In the following, the first communication node may include at least one of a base station and an emulator simulating a base station. The second communication node may include at least one of a terminal and an emulator simulating a terminal.

[0106] At least one of the first and second communication nodes (110, 120) can adjust a threshold value at which a certain function is triggered in the LCM in an AI / ML verification environment.

[0107] A threshold value may be used to trigger at least one of the following: a switch function of an AI / ML model (or feature, or model or training data associated with an associated ID), deactivation of AI / ML, fallback to Non-AI / ML, and AI / ML-related performance reporting function. For example, when the condition of the threshold value is satisfied, at least one of the following may be induced: a switch function of an AI / ML model (or feature, or model or training data associated with an associated ID), deactivation of AI / ML, fallback to Non-AI / ML, and AI / ML-related performance reporting function.

[0108] A threshold can represent the minimum performance standard for an AI / ML model. For example, if the performance metric of an AI / ML model falls below the threshold, the communication node may fall back to a legacy algorithm or deactivate the AI / ML.

[0109] Thresholds can serve as a criterion for determining pass or fail in conformance and post-deployment tests. For example, if the performance metrics of an AI / ML model fall below a threshold, the communication node may determine that the model failed the verification.

[0110] The threshold value may be used to determine the conditions under which at least one of the first and second communication nodes (110, 120) performs a management request, a management instruction, and a performance report. When the conditions of the threshold value are satisfied, at least one of the first and second communication nodes (110, 120) may perform at least one of a management request, a management instruction, and a performance report.

[0111] At least one of the first communication node and the second communication node can separately set a threshold value used in an AI / ML verification environment and a threshold value used in an AI / ML general operation environment.

[0112] The following describes the method for setting threshold values ​​used in AI / ML verification environments.

[0113] FIG. 7 is a flowchart illustrating an AI / ML verification method performed by a communication node according to an exemplary embodiment. In FIG. 7, the first communication node (110) may be referred to as a network, distinct from the terminal.

[0114] Referring to FIG. 7, in step S110, the first communication node (110) can transmit a terminal function request (UECapabilityEnquiry) message. The second communication node (120) can receive the terminal function request message. The second communication node (120) can collect information on supported AI / ML functions. The second communication node (120) can determine supported AI / ML functions by considering the hardware performance of the second communication node (120), the status of the AI / ML model, and whether inference support is possible. The AI / ML functions supported by the second communication node (120) can be indicated by at least one of an AI / ML model ID, an AI / ML function ID, and an associated ID. The second communication node (120) can configure terminal function information based on the information on supported AI / ML functions.

[0115] In step S120, the second communication node (120) can transmit a terminal capability information (UECapabilityInformation) message. The first communication node (110) can receive the terminal capability information message.

[0116] The first communication node (110) can generate first configuration information based on supported AI / ML function information included in terminal function information. The first configuration information may also be referred to as first RRC configuration information or first network configuration information.

[0117] The first configuration information may include network-side configuration information for implementing AI / ML functions that the second communication node (120) can support. For example, the first configuration information may include at least one of the following: information regarding AI / ML functions allowed by the network side among the AI / ML functions that the second communication node (120) can support, and a configuration regarding a message that the second communication node (120) must report to the network regarding the use of AI / ML functions.

[0118] In step S130, the second communication node (120) can transmit the first configuration information. The second communication node (120) can obtain the first configuration information.

[0119] In step S140, the second communication node (120) can determine applicable functionality based on the first configuration information. Here, 'applicable functionality' may include AI / ML functions available for communication between the first communication node (110) and the second communication node (120).

[0120] The second communication node (120) can determine applicable functions by considering information regarding AI / ML functions allowed by the network side and information regarding AI / ML functions that the second communication node (120) can support. The second communication node (120) can set a method for transmitting reporting messages to the first communication node (110) or the network while operating AI / ML functions based on the first configuration information.

[0121] In step S150, the second communication node (120) can transmit an applicable functionality report based on the applicable AI / ML functionality determined in step S140. For example, the second communication node (120) can transmit the applicable functionality report by including it in a UE Assistance message. As another example, the second communication node (120) can transmit the applicable functionality report through a message separate from the UE Assistance message. The first communication node (110) can receive the AI / ML functionality report.

[0122] In step S160, the first communication node (110) may transmit second configuration information. The second configuration information may also be referred to as first RRC configuration information or second network configuration information.

[0123] The first communication node (110) may include an inference configuration in the second configuration information. The first communication node (110) may transmit an inference configuration based on an applicable function report. The first communication node (110) may transmit an inference configuration required when using an AI / ML model corresponding to the applicable function. The second communication node (120) may obtain the inference configuration by receiving the second configuration information. The second communication node (120) may determine an inference method using AI / ML based on the inference configuration.

[0124] The above description explains the case where the inference configuration is included in the second configuration information. However, the embodiments are not limited thereto. For example, the inference configuration may be included in the first configuration information. In this case, the second communication node (120) may specifically determine an inference method using an AI / ML model corresponding to a function applicable in step S140. If the inference configuration is included in the first configuration information, step S160 may be omitted. However, even if the inference configuration is included in the first configuration information, if an update to the inference configuration occurs after step S150 is performed, the first communication node (110) may transmit the second configuration information.

[0125] The first communication node (110) can generate AI / ML configuration information based on information regarding the threshold value described above. The first communication node (110) can include AI / ML configuration information in the first configuration information transmitted at step S130. The threshold value included in the AI / ML configuration information can be used to trigger at least one of the switching function of the AI / ML model (or function, or model or training data associated with the associated ID), the deactivation of AI / ML, and the fallback function to Non-AI / ML, as described above.

[0126] A threshold can represent the minimum performance standard for an AI / ML model. For example, if the performance metric of an AI / ML model falls below the threshold, the communication node may fall back to a legacy algorithm other than AI / ML or deactivate the AI / ML. Therefore, the lower the threshold is set, the less frequently fallback, deactivation, switch, and report actions may occur. The higher the threshold is set, the more frequently fallback, deactivation, switch, and report actions may occur. However, the embodiments are not limited to this. Depending on the type of metric to which the threshold is applied, the manner in which the frequency of fallback, deactivation, switch, and report actions changes with changes in the threshold may vary. For example, if a smaller value of a specific metric indicates a higher performance standard, setting a lower threshold for that metric may increase the frequency of fallback, deactivation, and switch actions.

[0127] The first communication node (110) can set different threshold values ​​for the normal operation state of AI / ML and the verification state of AI / ML. For example, the first communication node (110) can apply a first threshold value in the normal operation state of AI / ML and apply a second threshold value in the verification state of AI / ML.

[0128] For example, the second threshold value may represent a performance standard lower than the first threshold value. For example, the second threshold value may be set smaller than the first threshold value.

[0129] The first communication node (110) can improve the efficiency and effectiveness of the AI / ML verification process by setting the second threshold value differently from the first threshold value. For example, by setting the second threshold value lower than the first threshold value, the intervention of stochastic variation effects occurring during the AI / ML verification process can be mitigated. AI / ML inference performance can fluctuate stochastically depending on the wireless environment, traffic conditions, etc. By applying a mitigated (low) threshold value during the AI / ML verification process, it is possible to prevent the verification from being interrupted or the conclusion from being distorted due to temporary performance degradation in the short term. It is possible to prevent premature return, deactivation, and switching actions during the AI / ML verification process. By reducing the frequency of return, deactivation, and switching actions, statistically significant verification of the average performance and stability of the AI / ML can be performed.

[0130] As another example, the second threshold value may represent a higher performance standard than the first threshold value. For instance, the second threshold value may be set higher than the first threshold value.

[0131] The second communication node (120) can perform fallback, deactivation, switch, and report operations when the AI / ML inference performance is higher than the second threshold or the first threshold. By setting the second threshold higher than the first threshold, the effects of probabilistic performance fluctuations occurring during the AI / ML verification process can be observed more efficiently. AI / ML inference performance may fluctuate probabilistically depending on the wireless environment, traffic conditions, etc. A second threshold with a higher value (indicating higher performance) may be applied during the AI / ML verification process. When the AI / ML performance is above the second threshold, the second communication node (120) performs at least one of fallback, deactivation, switch, or report operations to observe the performance range of the AI / ML model (function) under verification that is superior to the existing first threshold, thereby enabling efficient verification by identifying rapid performance improvement or delivering fewer report messages.

[0132] By setting the second threshold value differently from the first threshold value, the first communication node (110) and / or the second communication node (120) can acquire sufficient data during the AI / ML verification process and sufficiently capture under- / over-fit signals to utilize for fine-tuning and data augmentation decision-making. Additionally, the second threshold value different from the first threshold value can be used to omit unnecessary return, disable, switch-over actions, and unnecessary performance reporting and control procedures. This reduces signaling overhead by lowering the frequency of AI / ML-related reporting and control actions that occur during the AI / ML verification process.

[0133] By setting the second threshold lower than the first threshold, false positive results can be suppressed during the AI / ML verification process. Conversely, during normal operation, the first threshold is applied to reduce the risk of neglecting performance degradation and enable timely actions such as reverting, deactivating, and switching.

[0134] The first communication node (110) may include the first threshold value and the second threshold value in the first configuration information transmitted in step S130. If the first threshold value is shared in advance between the first communication node (110) and the second communication node (120) through another message, the first communication node (110) may include the second threshold value in the first configuration information.

[0135] The AI / ML configuration information included in the first configuration information may include identification information of an AI / ML model to which a first threshold and a second threshold are applied. The identification information of the AI / ML model may include at least one of an AI / ML model ID, an AI / ML function ID, and an associated ID. When identifying a model to be verified by unit of AI / ML model ID, the identification target can be subdivided and referred to. When identifying a model to be verified by unit of AI / ML function, AI / ML models associated with a specific function can be efficiently indicated and identified. When identifying a model to be verified by an associated ID, it may be easier to verify whether the training environment and the inference environment correspond in substance or are within a generalizable range. Through this, the verification subject can verify whether the AI / ML model is trained using a dataset that guarantees expected performance. If the first configuration information does not include a first threshold but includes a second threshold, the first configuration information may include identification information of an AI / ML model to which the second threshold is applied.

[0136] If the first configuration information includes a first threshold and / or a second threshold, the second communication node (120) may utilize the threshold set from the beginning when evaluating applicable functions in step S140. Additionally, in step S150, the second communication node (120) may generate and transmit applicable function reports by reflecting the preset threshold. That is, by presenting clear criteria from the beginning, consistency in AI / ML-related operations can be ensured and signaling overhead can be reduced.

[0137] The first communication node (110) may include AI / ML configuration information containing a first threshold and a second threshold in the second configuration information transmitted in step S160. If the first threshold is shared in advance between the first communication node (110) and the second communication node (120) through another message, the first communication node (110) may include AI / ML configuration information containing the second threshold in the second configuration information.

[0138] If the second configuration information includes a first threshold and / or a second threshold, in step S140, the second communication node (120) can determine applicable functions based on internal logic set by itself. Since the second configuration information is transmitted after steps S140 and S150 are performed, the first threshold and / or the second threshold can be set to reflect the final inference configuration and performance criteria. Thus, the first threshold and / or the second threshold can be set precisely and adaptively. The first threshold and / or the second threshold can be set to be more suitable for the actual communication environment.

[0139] The above explanation was based on the premise that the same metrics defined in both the AI / ML validation environment and the AI / ML operational environment are used. While the same metrics are used in both environments, the threshold values ​​applied to those metrics may be set differently in the two environments.

[0140] Metrics representing the performance of AI / ML may be defined differently in an AI / ML verification environment and an AI / ML normal operation environment. That is, the metrics used in the AI / ML verification environment and the metrics used in the AI / ML normal operation environment may be different. The first communication node (110) and / or the second communication node (120) may use distinct metrics in the AI / ML verification process and the AI / ML normal operation process, respectively. Since the purpose and method of performance monitoring in the AI / ML verification and AI / ML operation processes are different, using distinct metrics allows each procedure to be performed more efficiently.

[0141] AI / ML validation can be performed before an AI / ML model is deployed. AI / ML validation can also be performed after deployment when fine-tuning or model updates occur. AI / ML validation can be performed for the purpose of verifying the effectiveness and performance of the AI / ML model.

[0142] In the operation of AI / ML, performance monitoring is a continuous procedure performed in a real network environment and can be used to make decisions such as fallback, switch, retraining, and model adaptation.

[0143] Using the same metrics for both AI / ML verification and performance monitoring during operation can cause network congestion and signaling overhead, thereby reducing overall system efficiency. In particular, during post-deployment testing where AI / ML performance is being verified, the need for endpoints to participate in all aspects of LCM decision-making, such as rollback and changes, may be reduced.

[0144] As mentioned above, if the metrics and thresholds used in AI / ML operations are utilized as is in the AI / ML validation process, the intervention of stochastic performance variability effects becomes severe. Furthermore, actions such as rollbacks or modifications may be performed before the AI / ML model is sufficiently validated, making it difficult to fully verify the effects of fine-tuning or updates. Additionally, unnecessary signaling overhead may occur.

[0145] For example, the first communication node (110) and / or the second communication node (120) may use a first metric that is relatively lightweight and aggregated in the AI / ML normal operation procedure (performance monitoring procedure during AI / ML operation). The first metric used in the AI / ML normal operation procedure may have relatively low computational complexity. Therefore, the first communication node (110) and / or the second communication node (120) may use relatively few computational resources to calculate the first metric. The first metric may be expressed with a relatively small amount of data. For example, the first metric may include a confidence score, output deviation, trigger count, etc., but the embodiments are not limited thereto.

[0146] For example, the first communication node (110) and / or the second communication node (120) may use a relatively fine-grained second metric in the AI / ML verification procedure. The second metric used in the AI / ML verification procedure may have relatively high computational complexity. The second metric may be represented by a relatively large amount of data. Therefore, the first communication node (110) and / or the second communication node (120) may utilize relatively large computational resources to calculate the second metric. For example, the second metric may include inference accuracy, latency, loss function values, etc., but the embodiments are not limited thereto.

[0147] The first communication node (110) and / or the second communication node (120) can reduce signaling overhead by using a first metric that is relatively intensive and simplified in the AI / ML operation procedure. The first communication node (110) and / or the second communication node (120) can save computational resources used to calculate the first metric. Since the calculation and reporting of the first metric may occur relatively frequently in the AI / ML operation procedure, network efficiency can be improved by reducing signaling overhead and computational resource consumption through the use of a first metric that is relatively intensive and simplified.

[0148] In the AI / ML operation procedure, the frequency of generating and reporting the second metric may be relatively low. The first communication node (110) and / or the second communication node (120) can accurately evaluate the actual effect of model updates or fine-tuning by using a relatively precise and fine-grained second metric in the AI / ML verification procedure.

[0149] Although not shown in FIG. 7, the second communication node (120) can perform AI / ML verification based on the AI / ML configuration information obtained in step S130 or S160 of FIG. 7.

[0150] Figure 8 is a diagram showing the configuration method of IE included in AI / ML configuration information.

[0151] Referring to FIG. 8, the first configuration information may include AI / ML configuration information. The AI / ML configuration information is shown in R10.

[0152] The parameters shown in Fig. 8 are explained through Table 2.

[0153] Parameter Name Description validationMode Indicates whether AI / ML validation mode is being performed (e.g.) TRUE -> The terminal is performing AI / ML validation FALSE -> The terminal is performing normal AI / ML operations) fallbackSuppressionFlag Controls whether to suppress fallback or model switch operations triggered by LCM during the AI / ML validation phase (e.g.) TRUE -> Suppress fallback / switch during AI / ML validation FALSE -> Allow fallback / switch validationPerformanceMetric Specifies the secondary metric used in the AI / ML validation procedure (enumerated option) inferenceAccuracy: Accuracy of inference results (e.g., correct prediction vs. full prediction) latency: Time taken to complete one inference cycle lossFunction: Error calculated during validation (cross-entropy or MSE) otherMetric: Additional metrics not explicitly listed runtimeMonitoringMetric Specifies the primary metric used to monitor AI / ML model performance during normal operation (enumerated option) confidenceScore: Average model confidence of predictions Level (e.g., softmax output).outputDeviation: Deviation of the model output compared to the expected behavior.triggerCount: The number of times a monitoring event or specific metric has exceeded a threshold (e.g., fallback trigger).other: Placeholder for future or custom metrics.reportingThreshold: Defines a threshold (percentage or score) that determines when the terminal must report the first or second metric. The value of reportingThreshold depends on the metric selected in the first or second metric.fallbackTriggerThreshold: Specifies the threshold at which a fallback is triggered. The value of fallbackTriggerThreshold depends on the metric selected in the first or second metric.

[0154] Referring to FIG. 8 and Table 2, the AI / ML configuration information may include information on which metric each of the first metric and the second metric is defined as. The AI / ML configuration information may include a threshold at which a return is triggered and a threshold at which a report is triggered. In addition to those shown in FIG. 8 and Table 2, other thresholds may be further included in the configuration information for AI / ML verification. For example, the configuration information for AI / ML verification may include a threshold at which a switch is triggered or a threshold at which AI / ML deactivation is triggered.

[0155] The AI / ML configuration information may further include at least one of the information regarding the time point and time interval at which AI / ML verification is performed. The second communication node (120) can determine the time point for performing AI / ML verification based on at least one of the information regarding the time point and time interval at which AI / ML verification is performed, which is included in the AI / ML configuration information.

[0156] The second threshold value representing the above AI / ML performance criteria can be determined by comparing it with a measurement value reported in the form of a performance indicator (metric).

[0157] For example, in a Beam Management use case, the second communication node (120) reports a metric representing the number or ratio of accurately predicted reference signals within a predetermined window interval, and the network can use the number or ratio of accurately predicted reference signals to verify the performance of the AI / ML model by comparing it with a second threshold value. That is, the second metric may represent the number or ratio of reference signals that the second communication node (120) has predicted with an accuracy greater than a predetermined standard using the AI / ML model, and the second communication node (120) can verify the performance of the AI / ML model based on the comparison result between the second metric and the second threshold value.

[0158] The number of reference signals that the second communication node (120) predicts with an accuracy greater than a predetermined standard using an AI / ML model can correspond to the number of matches between the Top-M beams configured by the network (top M beams extracted based on L1-RSRP) and the Top-K beams (K beams predicted by the AI / ML model as having excellent reception performance) that the second communication node (120) predicts have excellent reception performance.

[0159] The ratio of reference signals predicted by the second communication node (120) using an AI / ML model with an accuracy greater than a predetermined standard for reception performance can be determined based on the ratio of overlap with the Top-M beam among the Top-K beams.

[0160] Accordingly, in a beam management use case, a measurement value (second metric) representing the number of reference signals (beams) predicted with an accuracy greater than a predetermined standard by the second communication node (120) may correspond to one example of an AI / ML performance indicator (measure value, metric) included in the comparison range of the second threshold standard of the present invention. Additionally, the network (i.e., the first communication node (110)) may configure multiple beams to improve the precision of verification (configuring multiple beams that the terminal must report), and the second communication node (120) may calculate the number of reference signals or the ratio value accurately predicted for each of the multiple beams by sharing and reusing the same measurement (L1-RSRP) and the same inference (Top-K) results. The second communication node (120) may package the composition of the measurement value (indicator, metric) reported for verification into a single synthetic report message and transmit it, or bundle multiple report compositions into a single transmission and report them. The second communication node (120) can report to the first communication node (110) the inference results for multiple reference signals (multiple beams) in a single report during the process of using an AI / ML model (function) based on AI / ML configuration information. Accordingly, multiple performance indicators are provided without additional measurement and inference, thereby reducing the overhead of message exchange and control signals required for reporting, as well as the computational load of the second communication node (120).

[0161] Figure 9 is a diagram showing an example of the implementation of AI / ML configuration information.

[0162] Referring to FIG. 9, since the validationMode in the first area (R22) is set to FALSE, the AI / ML configuration information described in the first area (R22) can be utilized when the AI / ML is operating normally. Since the validationMode in the second area (R32) is set to TRUE, the AI / ML configuration information described in the second area (R32) can be utilized in AI / ML verification.

[0163] The AI / ML configuration information described in the first area (R22) may include information regarding the first metric. For example, the first metric (runtimeMonitoringMetric) may be designated as the confidence score. The AI / ML configuration information described in the second area (R22) may include information regarding the second metric. For example, the second metric (validationPerformanceMetric) may be designated as inference accuracy.

[0164] The AI / ML configuration information described in the first area (R22) may include at least one threshold value related to the first metric. The AI / ML configuration information described in the second area (R32) may include at least one threshold value related to the second metric. The threshold value related to the first metric and the threshold value related to the second metric may each be set independently and differently from each other.

[0165] FIG. 10 is a diagram showing the configuration method of IE included in AI / ML configuration information. In describing the embodiment of FIG. 10, content that overlaps with FIG. 8 is omitted.

[0166] Referring to FIG. 10, the AI / ML configuration information may further include reportingLogic representing AI / ML-related reporting conditions. For example, reportingLogic may represent a comparison direction used to trigger the second communication node (120) to make AI / ML-related reports. That is, reportingLogic may represent signaling logic for AI / ML-related reports. When reportingLogic is set to reportAbove, the second communication node (120) may make an AI / ML-related report when a first metric or a second metric representing the performance of AI / ML is greater than a threshold value. When reportingLogic is set to reportBelow, the second communication node (120) may make an AI / ML-related report when a first metric or a second metric representing the performance of AI / ML is less than a threshold value.

[0167] FIG. 11 is a diagram showing an example of the implementation of AI / ML configuration information. In describing the embodiment of FIG. 11, content that overlaps with FIG. 9 is omitted.

[0168] Referring to FIG. 11, since the validationMode in the first area (R24) is set to FALSE, the AI / ML configuration information described in the first area (R24) can be utilized when the AI / ML is operating normally. Since the validationMode in the second area (R34) is set to TRUE, the AI / ML configuration information described in the second area (R34) can be utilized in AI / ML verification.

[0169] In the AI / ML configuration information described in the first area (R24), the reportingLogic can be set to reportAbove. Accordingly, when the AI / ML is operating normally, the second communication node (120) can perform AI / ML-related reporting when the confidenceScore is greater than the reportingThreshold of 85. The second communication node (120) can notify the first communication node (110) that a performance reporting triggering condition has occurred through the AI / ML-related reporting. The second communication node (120) can notify the first communication node (110) of information regarding at least one of the status of the AI / ML model and the performance of the AI / ML model through the AI / ML-related reporting.

[0170] In the AI / ML configuration information described in the second area (R34), the reportingLogic can be set to reportBelow. Accordingly, when the AI / ML is operating normally, the second communication node (120) can perform AI / ML-related reporting when the inference accuracy is less than the reporting threshold of 90.

[0171] As described above, the flexibility of communication network operation can be increased by using a parameter indicating a comparison direction used to trigger the second communication node (120) to make an AI / ML-related report.

[0172] For example, since the second metric used in the AI / ML verification process and the first metric used in the normal operation process of AI / ML are different, the reporting logic for each of the first and second metrics may be different. Therefore, the first communication node (110) can clearly control the reporting direction and reporting conditions by adding a reportingLogic field to the AI / ML configuration information.

[0173] The above describes the case where AI / ML configuration information is included in the first configuration information transmitted in step S130 of FIG. 7. However, the embodiments are not limited thereto. As with the description of the threshold value above, the AI / ML configuration information may also be included in the second configuration information.

[0174] After the deployment of the AI / ML model is completed, the AI / ML model may be updated (fine-tuned or updated) by the second communication node (120). Under current communication standards, there is no procedure to explicitly signal to the first communication node (110) even if the second communication node (120) updates the AI / ML model on its own. If the second communication node (120) updates the AI / ML model on its own and does not notify the first communication node (110), the network-side first communication node (110) may not be able to track or verify the updated model status. Furthermore, if the second communication node (120) assigns a new AI / ML model ID to the updated AI / ML model according to a predetermined rule or a rule it has set itself during the update process, information regarding the AI / ML model ID is not shared between the first communication node (110) and the second communication node (120), which may cause problems in future LCM full-cycle management.

[0175] According to the embodiment described below, the second communication node (120) can signal information about the update to the first communication node (110) after updating the AI / ML model. In particular, when post-deployment verification is performed network-driven, the signaling of the AI / ML model update can cause the first communication node (110) to perform post-deployment verification. Additionally, the signaling of the AI / ML model update can facilitate full-cycle management of the LCM by allowing update information regarding the AI / ML model to be shared between the first communication node (110) and the second communication node (120).

[0176] Figure 12 is a flowchart illustrating an AI / ML verification method that includes a process of transmitting a signaling indicating an AI / ML model update.

[0177] Referring to FIG. 12, in step S210, the second communication node (120) can perform an update of the AI / ML model. For example, the second communication node (120) can perform additional training or fine-tuning of the AI / ML model locally. For example, the second communication node (120) can perform an update of the AI / ML model based on the network, i.e., the association ID set by the first communication node (110). The second communication node (120) can perform an update of the AI / ML model by configuring training data or configuring a training environment with conditions corresponding to the pre-set association ID. The second communication node (120) can incrementally update the AI / ML model. The second communication node (120) can update the model by gradually adjusting or fine-tuning the existing AI / ML model parameters.

[0178] In step S220, the second communication node (120) may generate an AI / ML update report. The AI / ML update report may include identification information of the updated AI / ML model. The AI / ML update report may include at least one of an AI / ML model ID, an AI / ML function ID, and an association ID corresponding to the updated AI / ML model. The AI / ML update report may include a field (modelUpdateTimeStamp) regarding the time when the AI / ML model was updated. The AI / ML update report may include a field (modelUpdateReason) regarding the reason why the AI / ML model was updated.

[0179] The field regarding the time when the AI / ML model is updated may include more detailed information. For example, the field regarding the time when the AI / ML model is updated may include information regarding at least one of the time when the second communication node (120) updates the AI / ML model (function) and the time when the updated AI / ML model (function) is applied in the field by the second communication node (120).

[0180] In step S230, the second communication node (120) can transmit an AI / ML update report. The first communication node (110) can receive the AI / ML update report. In step S240, the first communication node (110) can transmit an ACK message indicating that it has received the AI / ML update report. The second communication node (120) can receive the ACK message.

[0181] The first communication node (110) can identify an updated AI / ML model based on the identification information of the AI / ML model included in the AI / ML update report. The first communication node (110) can identify an updated AI / ML model based on at least one of an AI / ML model ID, an AI / ML function ID, and an association ID. The first communication node (110) can check the data collection history or data change details corresponding to the association ID based on the association ID. The first communication node (110) can determine the AI / ML verification schedule by confirming the time when the AI / ML model was updated. The first communication node (110) can determine an appropriate AI / ML verification method and / or LCM full-cycle management method by confirming and analyzing the cause of the update. After synchronizing information regarding the AI / ML model management records with the second communication node (120), the first communication node (110) can perform the procedures necessary for AI / ML verification and LCM full-cycle management.

[0182] After the second communication node (120) updates the AI / ML model (function), the updated AI / ML model (function) may be applied based on the second communication node's (120) own judgment. In this case, as described above, the AI / ML update report may include information about the time when the updated AI / ML model (function) was applied.

[0183] As another example, after the second communication node (120) updates the AI / ML model (function), the updated AI / ML model (function) can be applied based on the judgment of the first communication node (110). That is, the first communication node (110) can determine the timing of the application of the updated AI / ML model (function). In this case, the AI / ML update report may not include information regarding the timing of the application of the updated AI / ML model (function). After step S230, the first communication node (110) can transmit information regarding the timing of the application of the updated AI / ML model (function) to the second communication node (120) through a separate signaling procedure.

[0184] With reference to FIGS. 1 to 12, an AI / ML verification method according to exemplary embodiments has been described above.

[0185] According to at least one embodiment, the threshold value used for AI / ML verification and the threshold value used for the normal operation of AI / ML can be set differently. This prevents unnecessary return operations during the AI / ML verification process and allows for the acquisition of sufficient data. Additionally, signaling overhead can be reduced during the AI / ML verification process.

[0186] According to at least one embodiment, distinct metrics may be used in the AI / ML validation environment and the AI / ML normal operation environment, respectively. By using distinct metrics, signaling overhead and metric computation resources can be reduced in the AI / ML normal operation. In AI / ML validation, the actual effects of model updates or fine-tuning can be accurately evaluated.

[0187] According to at least one embodiment, AI / ML configuration information may include a field representing signaling logic for AI / ML-related reporting. Flexibility in communication network operation may be increased by using a parameter representing a comparison direction used to trigger AI / ML-related reporting.

[0188] According to at least one embodiment, the second communication node can explicitly signal an update report after updating the AI / ML model. By signaling the update report, the latest information regarding the AI / ML model is shared between the first and second communication nodes, thereby facilitating AI / ML verification and LCM full-cycle management.

[0189] Based on the foregoing description of various embodiments of this disclosure, a person skilled in the art will clearly understand that the methods and / or processes of the present invention and the steps thereof may be realized in hardware, software, or any combination of hardware and software suitable for a particular use case. The hardware may include a general-purpose computer and / or a dedicated computing device or a specific computing device or a particular form or component of a specific computing device. The processes may be realized by one or more processors having internal and / or external memory, such as a microprocessor, a controller, such as a microcontroller, an embedded microcontroller, a microcomputer, an arithmetic logic unit (ALU), a digital signal processor, such as a programmable digital signal processor, or other programmable device. In addition, or as an alternative, the above processes may be carried out by an application-specific integrated circuit (ASIC), a programmable gate array, such as a field programmable gate array (FPGA), a programmable logic unit (PLU), or a programmable array logic (PAL), or any other device capable of executing and responding to instructions, any other device or combination of devices that may be configured to process electronic signals. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software.For ease of understanding, the processing unit may be described as being used as a single unit, but a person of ordinary skill in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0190] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more machine-readable recording media.

[0191] Furthermore, parts contributing to the objects of the technical solution of the present invention or to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-readable medium. The machine-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the machine-readable recording medium may be those specifically designed and configured for the embodiments, or they may be those known and available to a person skilled in the art of computer software. Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-rays; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, bytecode, as well as high-level language code that can be executed by a computer using an interpreter, etc., which can be created using a structured programming language such as C, an object-oriented programming language such as C++, or high-level or low-level programming languages ​​(assembly language, hardware description languages, and database programming languages ​​and technologies), which can be stored and compiled or interpreted to be executed on a machine capable of executing any of the aforementioned devices, as well as a processor, a processor architecture, or a heterogeneous combination of different hardware and software combinations.

[0192] Accordingly, in one embodiment according to the present invention, when the methods and combinations thereof described above are performed by one or more computing devices, the methods and combinations thereof may be implemented as executable code that performs each step. In another embodiment, the methods may be implemented as systems that perform the steps, and the methods may be distributed in various ways across devices, or all functions may be integrated into a single dedicated, standalone device or other hardware. In yet another embodiment, the means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

[0193] For example, the above-described hardware device may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, or TPU that is combined with memory such as ROM / RAM for storing program instructions and configured to execute instructions stored in said memory, and may include a communication unit capable of exchanging signals with an external device. Additionally, the hardware device may include a keyboard, mouse, or other external input device for receiving instructions written by developers.

[0194] Although the present invention has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments, and a person skilled in the art to which the invention belongs can make various modifications and variations from this description.

[0195] Accordingly, the scope of the present invention is not limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims attached to this disclosure, as well as the claims attached to this disclosure, shall be considered to be within the scope of the scope of the concept of the present invention. For example, appropriate results may be achieved even if the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from the described method, or are replaced or substituted by other components or equivalents.

[0196] Such equivalent or equivalent modifications may include, for example, logically equivalent methods capable of producing the same result as carrying out the method according to the present invention; the true meaning and scope of the present invention shall not be limited by the examples described above, but shall be understood in the broadest sense permissible by law.

Claims

1. An AI / ML verification method performed by a first communication node including a communication unit and a processor, wherein The method includes the step of generating AI / ML configuration information including a first threshold value representing an AI / ML performance standard applied to the normal operation of AI / ML and a second threshold value representing an AI / ML performance standard applied to the verification of a distinguished AI / ML, and transmitting said AI / ML configuration information. The above second threshold value is an AI / ML verification method comprising a threshold value for determining a condition in which at least one of returning to non-AI / ML (Non-AI / ML), deactivating AI / ML, changing an AI / ML model, and reporting AI / ML-related data occurs during AI / ML verification.

2. In Paragraph 1, An AI / ML verification method in which the second threshold value is set to represent a relatively lower AI / ML performance standard compared to the first threshold value.

3. In Paragraph 1, An AI / ML verification method in which the second threshold value is set to represent a relatively higher AI / ML performance standard compared to the first threshold value.

4. In Paragraph 1, An AI / ML verification method in which the second threshold value is set such that, compared to the first threshold value, the range in which conditions are satisfied such that at least one of a return to non-AI / ML, deactivation of AI / ML, change of AI / ML model, and AI / ML-related reporting occurs in the verification of AI / ML is reduced.

5. In Paragraph 1, The above AI / ML configuration information An AI / ML verification method comprising instruction information for a first metric representing AI / ML performance in normal operation of AI / ML, and instruction information for a second metric representing AI / ML performance in verification of AI / ML, which is set separately from the first metric.

6. In Paragraph 5, The above second metric is an AI / ML verification method with relatively higher computational complexity compared to the above first metric.

7. In Paragraph 5, The above second metric is an AI / ML validation method expressed with a relatively large amount of data compared to the above first metric.

8. In Paragraph 5, The first metric above includes at least one of a confidence score for an AI / ML model, an output deviation, and the number of trigger occurrences, and The above second metric is an AI / ML validation method comprising at least one of the inference accuracy, latency, and loss function values ​​of an AI / ML model.

9. In Paragraph 1, The above AI / ML configuration information An AI / ML verification method comprising at least one of information indicating a signaling logic for AI / ML-related reporting in normal operation of AI / ML and information indicating a signaling logic for AI / ML-related reporting in verification of AI / ML.

10. In Paragraph 9, An AI / ML verification method in which the signaling logic for AI / ML-related reporting in normal operation of the above AI / ML and the signaling logic for AI / ML-related reporting in verification of the above AI / ML each indicate a comparison direction used to trigger AI / ML-related reporting.

11. In Paragraph 1, A step of transmitting a terminal function request message to a second communication node; A step of obtaining terminal function information from the second communication node; A step of transmitting first configuration information to the second communication node including AI / ML function information allowed by the first communication node based on the terminal function information; and The method includes the step of obtaining an applicable AI / ML function report from the second communication node, The above AI / ML configuration information is an AI / ML verification method included in the above first configuration information.

12. In Paragraph 1, A step of transmitting a terminal function request message to a second communication node; A step of obtaining terminal function information from the second communication node; A step of transmitting first configuration information to the second communication node, which includes AI / ML function information allowed by the first communication node based on the terminal function information; A step of obtaining an applicable AI / ML function report from the second communication node; and The method includes the step of transmitting second configuration information including an inference configuration based on the above applicable AI / ML function report, and The above AI / ML configuration information is an AI / ML verification method included in the above second configuration information.

13. An AI / ML verification method performed by a second communication node including a communication unit and a processor, A step of obtaining AI / ML configuration information from a first communication node, including a first threshold value representing an AI / ML performance criterion applied to the normal operation of AI / ML and a second threshold value representing an AI / ML performance criterion applied to the verification of AI / ML distinguished from it; and It includes a step of performing AI / ML verification based on the above AI / ML configuration information, The above second threshold value is an AI / ML verification method comprising a threshold value for determining a condition in which at least one of returning to non-AI / ML (Non-AI / ML), deactivating AI / ML, changing an AI / ML model, and reporting AI / ML-related data occurs during AI / ML verification.

14. In Paragraph 13, An AI / ML verification method in which the second threshold value is set to represent a relatively lower AI / ML performance standard compared to the first threshold value.

15. In Paragraph 13, The above AI / ML configuration information An AI / ML verification method comprising instruction information for a first metric representing AI / ML performance in normal operation of AI / ML, and instruction information for a second metric representing AI / ML performance in verification of AI / ML, which is set separately from the first metric.

16. In Paragraph 13, The above AI / ML configuration information An AI / ML verification method comprising at least one of information indicating a signaling logic for AI / ML-related reporting in normal operation of AI / ML and information indicating a signaling logic for AI / ML-related reporting in verification of AI / ML.

17. In Paragraph 13, Step to perform updates to the AI / ML model; A step of generating an AI / ML update report based on the update information of the above AI / ML model; and AI / ML verification method comprising the step of transmitting the above AI / ML update report.

18. In Paragraph 17, The above AI / ML update report is An AI / ML verification method comprising at least one of identification information of an updated AI / ML model, information on the time at which the AI / ML model was updated, and information on the cause of the update of the AI / ML model.

19. In Paragraph 18, The above AI / ML update report is an AI / ML verification method that includes information about the time when the updated AI / ML model is applied by the second communication node.

20. In Paragraph 18, An AI / ML verification method in which the identification information of the above-mentioned updated AI / ML model is indicated by at least one of an AI / ML model ID, an AI / ML function ID, and an association ID.

Images