Method and device for ai / ML verification in wireless communication system
The method verifies AI/ML models in wireless communication systems by evaluating the relationship between measurement and prediction beams, addressing performance uncertainty and reducing computational complexity.
Patent Information
- Application Number
- PCT/KR2026/096786
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-22
- Filing Date
- 2026-05-11
- Publication Date
- 2026-11-12
AI Technical Summary
There is no guarantee that the performance of AI/ML models will improve after deployment in wireless communication systems, necessitating effective verification and testing to ensure their functionality and performance.
A method for verifying AI/ML in wireless communication systems by deriving measurement reference beams and prediction candidate beams, evaluating their relationship, and generating a verification report to assess beam management performance, which includes multiple measurement criteria for comprehensive evaluation.
This method allows for the effective verification of AI/ML-based beam management performance, reducing overestimation of model performance and alleviating computational complexity while ensuring high-quality beam prediction.
Smart Images

Figure KR2026096786_12112026_PF_FP_ABST
Abstract
Description
AI / ML verification method and device in a wireless communication system
[0001] The present disclosure relates to a method for verifying AI / ML in a wireless communication system, and more specifically, to an AI / ML management method capable of verifying / testing the functions, performance, etc. of AI / ML in a wireless communication system that supports AI / ML.
[0002] A communication system may include a core network, base stations (e.g., macro base stations, small base stations, relays, etc.), terminals, etc. Communication between a base station and a terminal may be performed based on various radio access technologies (RATs) (e.g., 4G communication technology, 5G communication technology, 6G communication technology, WLAN (wireless local area network) technology, WPAN (wireless personal area network) technology, etc.).
[0003] 3GPP selected AI / ML as a Study Item for 5G Release 18 and conducted research and discussions. AI / ML technology utilizes artificial intelligence and machine learning to streamline network operations and optimize service quality. 3GPP is prioritizing the introduction of AI / ML in scenarios such as 5G beam management, positioning, channel state prediction, and channel state compression, expecting improvements in system throughput, resource utilization, and coverage.
[0004] In addition, 3GPP has begun full-scale 6G research and is considering AI / ML as the core of the AI-native architecture for 6G. Through this, based on AI / ML frameworks developed in 5G, it is researching data management, model deployment, and Air Interface optimization (e.g., CSI compression, mobility enhancement), and aims to enable autonomous AI operations, maximize network efficiency, and support new use cases in 6G systems.
[0005] Standardization organizations such as 3GPP and the O-RAN Alliance are defining a Life Cycle Management (LCM) framework for AI / ML functions to apply AI / ML to various network functions and to operate them efficiently.
[0006] LCM includes a series of management procedures such as deployment, enabling, performance monitoring, disabling, and removal of AI / ML models. LCM also includes features to replace the model with existing logic-based functions, switch to another AI / ML model with better performance, or fallback to a legacy system that does not utilize AI / ML if the model's performance falls below a predefined threshold.
[0007] AI / ML models can be modified or updated even after deployment. Additionally, new AI / ML models may be deployed. There is no guarantee that the performance of the device will improve during use even after deployment or updates. Therefore, it is essential to perform verification and testing of the AI / ML when applying new or updated AI / ML models to communication nodes.
[0008] According to at least one embodiment, a method and apparatus for verifying AI / ML are disclosed.
[0009] An AI / ML verification method is disclosed in a wireless communication system performed by a second communication node comprising a communication unit and a processor. The disclosed method comprises the steps of: obtaining AI / ML verification configuration information from a first communication node; deriving a set of measurement reference beams corresponding to each of a plurality of measurement reference counts M based on beam results measured at at least one time instance; deriving a set of prediction candidate beams corresponding to a number of prediction candidates K based on AI / ML inference results at at least one time instance; for each of the plurality of measurement reference counts M, determining a verification result for each M value by evaluating the relationship between the set of measurement reference beams corresponding to the measurement reference count M and the set of prediction candidate beams; and generating a verification report on AI / ML-based beam management performance based on the verification result for each M value, wherein the AI / ML verification configuration information may include information regarding the values of the plurality of measurement reference counts M.
[0010] According to at least one embodiment, AI / ML-based beam management performance can be verified based on the relationship between the Top-K predicted candidate beam set derived from AI / ML inference results and the Top-M measurement reference beam set derived from beam measurement results. Accordingly, it is possible to directly evaluate whether the AI / ML model includes high-quality measurement beams in the predicted candidate beam set in an actual wireless environment.
[0011] According to at least one embodiment, a verification result can be produced based on the intersection, inclusion relationship, overlap relationship, or the number of elements in the intersection between the Top-K prediction candidate beam set and the Top-M metric beam set. Accordingly, AI / ML-based beam prediction performance, which is difficult to express solely through a comparison method of simple scalar values and thresholds, can be effectively verified based on the relationships between beam sets.
[0012] According to at least one embodiment, verification results for each M value can be produced for a plurality of measurement criteria M values. Accordingly, the performance of an AI / ML model can be assessed stepwise at different measurement criterion widths or verification granularity levels.
[0013] According to at least one embodiment, unlike the case where only one large M value is used, the verification results for a small M value and the verification results for a large M value can be checked together. Accordingly, it is possible to distinguish whether the AI / ML model includes the actual top measurement beam within the prediction candidate or includes a valid beam among a wider set of measurement candidate beams within the prediction candidate, and the overestimation of the performance of the AI / ML model can be reduced.
[0014] According to at least one embodiment, at least one of the AI / ML inference results and beam measurement results obtained in the same or interrelated time instances can be reused for verification of multiple M values. Accordingly, even when verification of multiple M values is performed, additional AI / ML inference or additional beam measurements can be reduced, and the computational complexity and measurement burden of the second communication node can be alleviated.
[0015] According to at least one embodiment, verification results for a plurality of M values can be included in a single composite verification report. Accordingly, the number of reporting procedures, the number of reconstruction procedures, or the number of signaling transactions can be reduced compared to cases where verification results for each M value are requested or reported individually.
[0016] FIG. 1 is a drawing showing a wireless communication system (100) according to an exemplary embodiment.
[0017] FIG. 2 is a block diagram illustrating the configuration of a communication node (200) that constitutes a communication system.
[0018] Figure 3 is a diagram illustrating an exemplary 5G network architecture.
[0019] Figure 4 is a diagram showing a 6G network architecture.
[0020] Figure 5 is a block diagram showing the full cycle of AI / ML functions in a mobile communication system.
[0021] Figure 6 is a diagram showing the prediction error according to the training of an AI / ML model.
[0022] FIG. 7 is a flowchart illustrating an AI / ML verification method performed by a communication node according to an exemplary embodiment.
[0023] Figure 8 is a diagram illustrating an exemplary configuration method of IE included in AI / ML configuration information.
[0024] Figure 9 is a diagram showing an example of the implementation of AI / ML configuration information.
[0025] Figure 10 is a diagram illustrating an exemplary configuration method of IE included in AI / ML configuration information.
[0026] Figure 11 is a diagram showing an example of the implementation of AI / ML configuration information.
[0027] Figure 12 is a flowchart illustrating an AI / ML verification method that includes a process of transmitting a signaling indicating an AI / ML model update.
[0028] FIG. 13 is a flowchart illustrating an AI / ML verification method performed by a second communication node according to an exemplary embodiment.
[0029] Figure 14 is a conceptual diagram structurally illustrating the AI / ML verification method described with reference to Figure 13.
[0030] Figure 15 is a block diagram showing the configuration of a verification report.
[0031] FIG. 16 illustrates an AI / ML verification method performed by a first communication node.
[0032] FIG. 17 illustrates a local verification and model update reporting procedure performed by a second communication node in an FL / DL environment.
[0033] FIG. 18 illustrates a structure that synchronizes local model status and global model status in an FL / DL environment and performs FL / DL-based AI / ML model management operations based on status and KPI reporting.
[0034] An AI / ML verification method is disclosed in a wireless communication system performed by a second communication node comprising a communication unit and a processor. The disclosed method comprises the steps of: obtaining AI / ML verification configuration information from a first communication node; deriving a set of measurement reference beams corresponding to each of a plurality of measurement reference counts M based on beam results measured at at least one time instance; deriving a set of prediction candidate beams corresponding to a number of prediction candidates K based on AI / ML inference results at at least one time instance; for each of the plurality of measurement reference counts M, determining a verification result for each M value by evaluating the relationship between the set of measurement reference beams corresponding to the measurement reference count M and the set of prediction candidate beams; and generating a verification report on AI / ML-based beam management performance based on the verification result for each M value, wherein the AI / ML verification configuration information may include information regarding the values of the plurality of measurement reference counts M.
[0035] The above AI / ML validation configuration information further includes information regarding the number of prediction candidates K, and the number of prediction candidates K can be applied as a fixed value regardless of the number of multiple measurement criteria M.
[0036] The number of predicted candidates K mentioned above can be set for each of the plurality of measurement criteria M based on the corresponding number of measurement criteria M.
[0037] At least one of the AI / ML inference result and the beam measurement result obtained at each of the above at least one time instance can be reused to derive at least one of the measurement reference beam set for each of the above prediction candidate beam set and the above plurality of measurement reference count M.
[0038] The above verification report can be generated as a composite verification report including verification results for each M value corresponding to each of the plurality of measurement criteria M.
[0039] The verification result for each M value above can be determined based on at least one of the intersection, inclusion relationship, overlap relationship between the measurement reference beam set corresponding to the number of measurement references M and the prediction candidate beam set, and the number of elements in the intersection.
[0040] The verification result for each value of M above can be determined based on whether the intersection of the set of measurement beams corresponding to the number of measurement criteria M at a specific time instance and the set of prediction candidate beams is an empty set.
[0041] The verification result for each value of M above can be determined based on whether the number of elements in the intersection of the set of measurement beams corresponding to the number of measurement criteria M in a specific time instance and the set of prediction candidate beams is greater than or equal to the threshold value.
[0042] The above verification results for each M value may include the number of elements in the intersection of the set of measurement beams corresponding to the number of measurement criteria M at a specific time instance and the set of prediction candidate beams.
[0043] The verification result for each value of M above can be determined based on at least one of the number and ratio of time instances where the intersection of the set of measurement reference beams corresponding to the number of measurement references M among a plurality of time instances and the set of prediction candidate beams is not an empty set.
[0044] The verification result for each value of M above can be determined based on the number of time instances where the number of elements in the intersection of the measurement reference beam set corresponding to the number of measurement references M among multiple time instances and the prediction candidate beam set is greater than or equal to the reference value, or matches the reference value.
[0045] The verification result for each of the above M values may be determined based on at least one of the sum of the number of elements in the intersection of the measurement beam set corresponding to the number of measurement criteria M determined at each of the plurality of time instances and the prediction candidate beam set, the average, and the ratio of the normalized number of elements in the intersection.
[0046] A second communication node comprising a communication unit and a processor is disclosed. The processor performs the following steps: obtaining AI / ML verification configuration information from a first communication node; deriving a set of measurement reference beams corresponding to each of a plurality of measurement reference counts M based on beam results measured at at least one time instance; deriving a set of prediction candidate beams corresponding to a number of prediction candidates K based on AI / ML inference results at at least one time instance; for each of the plurality of measurement reference counts M, determining a verification result for each M value by evaluating the relationship between the set of measurement reference beams corresponding to the measurement reference count M and the set of prediction candidate beams; and generating a verification report on AI / ML-based beam management performance based on the verification result for each M value, wherein the AI / ML verification configuration information may include information regarding the values of the plurality of measurement reference counts M.
[0047] A method for verifying AI / ML in a wireless communication system is disclosed, which is performed by a first communication node including a communication unit and a processor. The disclosed method comprises the steps of: providing AI / ML verification configuration information to a second communication node; receiving a verification report on AI / ML-based beam management performance from the second communication node; and performing AI / ML operation control based on the verification report. The AI / ML verification configuration information includes information on a plurality of measurement reference count M values, and the verification report is generated based on a verification result for each M value corresponding to each of the plurality of measurement reference count M. The verification result for each M value can be determined by evaluating the relationship between a set of prediction candidate beams corresponding to a number of prediction candidates K derived based on AI / ML inference results in at least one time instance, and a set of measurement reference beams corresponding to a number of measurement reference M derived based on beam results measured in at least one time instance.
[0048] The above AI / ML operation control may include at least one of determining the validation result of an AI / ML model, managing the lifecycle of an AI / ML model, returning to a non-AI / ML state, disabling AI / ML functions, and reconfiguring AI / ML validation configuration information.
[0049] The above AI / ML validation configuration information further includes information regarding the number of prediction candidates K, and the number of prediction candidates K can be applied as a fixed value regardless of the number of multiple measurement criteria M.
[0050] The above verification report may be a composite verification report containing verification results for each M value corresponding to each of the plurality of measurement criteria M.
[0051] The verification result for each M value above can be determined based on at least one of the intersection, inclusion relationship, overlap relationship between the measurement reference beam set corresponding to the number of measurement references M and the prediction candidate beam set, and the number of elements in the intersection.
[0052] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.
[0053] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.
[0054] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0055] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0056] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0057] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate an overall understanding of the present invention, the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.
[0058] Table 1 shows the abbreviations used in the present disclosure.
[0059] 약어전체 이름AI / MLArtificial Intelligence / Machine Learning3GPP3rd Generation Partnership ProjectACKAcknowledgementAFApplication FunctionAIArtificial IntelligenceAMFAccess and Mobility Management FunctionAUSFAuthentication Server FunctionBWPBandwidth PartC-RNTICell RNTICSIChannel State InformationCSI-RSChannel State Information Reference SignalCLICross link InterferenceCEControl ElementDCIDownlink Control InformationIEInformation elementMACMedium Access ControlMLMachine LearningNSSFNetwork Slicing Selection FunctionNEFNetwork Exposure FunctionNRFNF Repository FunctionPCFPolicy Control FunctionPDCCHPhysical Downlink Control ChannelPDSCHPhysical Downlink Shared ChannelPDUProtocol Data UnitPRACHPhysical Random Access ChannelPT-RSPhase Tracking Reference SignalPUCCHPhysical Uplink Control ChannelPUSCHPhysical Uplink Shared ChannelRARandom AccessRACHRandom Access ChannelRANRadio Access NetworkRBResource BlockRRCRadio Resource ControlRSRPReference Signal Received PowerRSRQReference Signal ReceivedQualityRSSIReceived Signal Strength IndicatorSBFDSub-Band Full DuplexSISelf InterferenceSIBSystem information blockUEUser EquipmentDLDownlinkULUplinkRARRandom Access ResponseRRMRadio Resource ManagementRRCRadio Resource ControlRSRPReference Signal Received PowerQoSQuality of ServiceDRBData Radio BearerDAPSDual Active Protocol StackTDDTime Division DuplexingFDDFrequency Division DuplexingSSBSS BlockCQIChannel Quality IndicatorPMIPrecoding Matrix IndicatorLILayer IndicatorLCMLife Cycle ManagementRIRank IndicatorQCLQuasi coLocation
[0060] In the following, "Verification of AI / ML" refers to a procedure for evaluating or testing the performance of AI / ML models identified by at least one of a predetermined AI / ML model ID, an AI / ML function ID, and an associated ID. Verification of AI / ML may include verification procedures conducted in a predetermined test environment simulating a commercial communication network and verification procedures conducted on a commercial communication network. Verification of AI / ML may include at least one of a conformance test performed in advance before the AI / ML model is applied to the field and a post-deployment test performed after the AI / ML model is applied to the field.
[0061] FIG. 1 is a drawing showing a wireless communication system (100) according to an exemplary embodiment.
[0062] Referring to FIG. 1, a wireless communication system (100) may be composed of a plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4). Here, a communication node refers to a node capable of transmitting and receiving signals in the wireless communication system (100), and each of the plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) may support at least one communication protocol. For example, each of the multiple communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) can support cellular communication (e.g., LTE (long term evolution), LTE-A (advanced), 5G NR, 5G-Advanced, etc. as defined in the 3GPP (3rd generation partnership project) standard).
[0063] For example, each of the multiple communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) can support a CDMA (code division multiple access) based communication protocol, a WCDMA (wideband CDMA) based communication protocol, a TDMA (time division multiple access) based communication protocol, a FDMA (frequency division multiple access) based communication protocol, an OFDM (orthogonal frequency division multiplexing) based communication protocol, an OFDMA (orthogonal frequency division multiple access) based communication protocol, a SC (single carrier)-FDMA based communication protocol, a NOMA (non-orthogonal multiple access) based communication protocol, a SDMA (space division multiple access) based communication protocol, a SBFD (sub-band full duplex), AI / ML, etc.
[0064] A plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) may include a plurality of base stations (110-1, 110-2, 110-3) and a plurality of terminals (120-1, 120-2, 120-3, 120-4). Each of the base stations (110-1, 110-2, 110-3) may form a cell. The cell may include a small cell, a macro cell, a pico cell, a femto cell, etc., but the embodiments are not limited thereto. For example, the first terminal (120-1) and the second terminal (120-2) may be within the coverage of the first base station (110-1), the third terminal (120-3) may be within the coverage of the second base station (110-2), and the fourth terminal (120-4) may be within the coverage of the third base station (110-3). Additionally, some of the multiple terminals (120-1, 120-2, 120-3, 120-4) may support AI / ML functions, while others may not support AI / ML functions. 3GPP is conducting technology research in 6G, including new interfaces, advanced sensing, AI / ML-based automation, and sustainable connectivity.
[0065] Multiple base stations (110-1, 110-2, 110-3) may be referred to as gNodeB (gNB), NodeB, evolved NodeB, BTS (base transceiver station), radio base station, radio transceiver, access point, access node, roadside unit (RSU), RRH (radio remote head), TP (transmission point), TRP (transmission and reception point), relay node, etc. Multiple base stations (110-1, 110-2, 110-3) may form a Radio Access Network (RAN). The Radio Access Network may be connected to a core network.
[0066] Each of the multiple terminals (120-1, 120-2, 120-3, 120-4) may be referred to as a terminal, access terminal, mobile terminal, station, subscriber station, mobile station, portable subscriber station, node, device, etc.
[0067] The communication protocol support ranges of each of the multiple base stations (110-1, 110-2, 110-3) may differ from one another. For example, some of the multiple base stations (110-1, 110-2, 110-3) may support AI / ML functions while others do not. Similarly, the communication protocol support ranges of each of the terminals (120-1, 120-2, 120-3, 120-4) may also differ from one another. For example, some of the multiple terminals (120-1, 120-2, 120-3, 120-4) may support AI / ML functions while others do not.
[0068] FIG. 2 is a block diagram illustrating the configuration of a communication node (200) that constitutes a communication system. At least some of the communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3) shown in FIG. 1 may correspond to the communication node (200) shown in FIG. 2.
[0069] Referring to FIG. 2, the communication node (200) may include at least one processor (210), a memory (220), and a transceiver (230) that is connected to a network to perform communication. Additionally, the communication node (200) may further include an input interface device (240), an output interface device (250), a storage device (260), etc. Each component included in the communication node (200) may be connected by a bus (270) to communicate with one another.
[0070] The processor (210) can execute a program command stored in at least one of the memory (220) and the storage device (260). The processor (210) may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Each of the memory (220) and the storage device (260) may be composed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (220) may be composed of at least one of read-only memory (ROM) and random access memory (RAM).
[0071] FIG. 3 is a diagram illustrating an exemplary 5G network architecture. Referring to FIG. 3, the 5G network architecture may include an NSSF providing a network slicing control solution, an NEF providing a network function opening solution, an NRF providing an NF interoperability control solution within the 5G network, a PCF providing a data packet flow policy control solution, a UDM providing a user information and policy management solution, an AMF providing a network access and mobility management solution, an SMF providing a terminal / network session management solution, a UPF providing a solution for user packet routing and terminal connectivity between base stations, a radio access network (RAN), and a user terminal (UE).
[0072] N1 reference point is defined to transmit signaling between the UE and the AMF, N2 is defined as the reference point to connect the RAN access node and the AMF, and N3 is defined as the reference point to connect the RAN access node and the UPF.
[0073] 3GPP is currently conducting 6G research, and the 6G network architecture plans to integrate AI-native design, advanced sensing, and ultra-low latency communication based on the existing 5G structure. In 6G, the N4 interface between the User Plane Function (UPF) and Session Management Function (SMF) and the N3 interface between the RAN and UPF will be optimized, and AI / ML-based dynamic resource management and network slicing will be enhanced. Furthermore, 6G is highly likely to support industrial use cases (e.g., autonomous driving, holographic communication) by introducing new Network Functions (NFs) and interfaces.
[0074] Figure 4 is a diagram showing a 6G network architecture.
[0075] Referring to FIG. 4, the 6G network architecture provides AI-based end-to-end network management and automation services and analysis information exposure, and provides support functions based on cloud infrastructure, transmission network, data pipeline, and common platform functions. The 6G network architecture includes a 6G terminal (610), a 6G radio access network (620, RAN), and an AI native cloud-based 6G core network (630, CN), and can also be interconnected with an existing 5G network as needed.
[0076] Here, the opening of AI-based service and analysis information supports the improvement of network efficiency and service quality by providing AI-processed network status information and analysis results to service providers or external systems. In addition, 6G core networks operating in a cloud-native environment enable dynamic network resource management, autonomous service orchestration, and efficient user service quality assurance through network functions with embedded artificial intelligence (AI).
[0077] This architecture strengthens the connection with the data network to provide flexibility in data flow management and defines an efficient communication path and management interface between the 6G terminal (610) and the data network. In particular, the AI-based E2E end-to-end management function enables autonomous and intelligent network control across the entire network area (terminal-RAN-CN-data network).
[0078] FIG. 5 is a block diagram illustrating the full lifecycle of AI / ML functions in a mobile communication system. Operations such as updating, changing, managing, verifying, and reporting of AI / ML-related models described above can all be performed within the full lifecycle management (LCM) framework shown in FIG. 5. The AI / ML function blocks shown in FIG. 5 can be implemented by at least one of the communication nodes of the mobile communication system. For example, the AI / ML function blocks shown in FIG. 5 can be implemented by at least one of a base station and a terminal.
[0079] When the first terminal (120-1) is connected to the service area of the first base station (110-1) that supports AI / ML functions, the first terminal (120-1) or the first base station (110-1) can activate at least some of the function blocks (410, 412, 414, 416, 418) of FIG. 5.
[0080] The terminal (120-1) can collect data related to the wireless channel by activating the data collection block (410). The terminal (120-1) can collect data related to wireless channel information from reference signals (CSI-RS, SSB, PT-RS, PRS, downlink reference signals, etc.). When the base station (110-1) activates the data collection block (410), it can collect data related to wireless channel information from reference signals (SRS, PRS, PT-RS) and channel status reports, etc.
[0081] Data collected from the data collection block (410) can be used as data for training / monitoring / inference. The model training block (412) can train an AI / ML model using the data collected from the data collection block (410). The model storage block (418) can store the trained AI / ML model. The stored model can be used by the inference block (416). The inference block (416) can infer information necessary for the operation of a wireless communication system based on the data collected from the data collection block (410).
[0082] The inference block (416) can report the inference results to the AI / ML management block (414). The AI / ML management block (414) can perform overall functions related to training the AI / ML model, selecting the AI / ML model, enabling / disabling the AI / ML model, and controlling other function blocks. The AI / ML management block (414) can monitor the performance of the AI / ML model to maintain the performance of the AI / ML and provide feedback on the training results to the model training block (412). If the performance of the monitored AI / ML model is low and retraining is required, the AI / ML management block (414) can request retraining from the model training block (412). The AI / ML management block (414) can determine the enabling / disabling of the AI / ML model and the selection / change of the model. The AI / ML management block (414) performs management instructions such as a fallback to an existing communication method that does not use AI / ML functions.
[0083] AI / ML models can be classified in various ways. For example, a model ID can be assigned to each AI / ML model. As another example, AI / ML models can be classified by function, and a functionality ID can be assigned to each function. Associated IDs may also be used to distinguish AI / ML models.
[0084] An Associated ID may represent a configuration for data collection to train an AI / ML model. An Associated ID can be used to ensure the identity between the training environment and the inference environment of the AI / ML model. A terminal (120-1) can verify the configuration for data collection from the Associated ID and collect data according to that configuration. The terminal (120-1) can train or develop an AI / ML model using the collected data based on the Associated ID.
[0085] AI / ML models possess the characteristics of overfitting and generalization. Overfitting refers to a situation where a model performs excellently only on a specific dataset but degrades on others. Conversely, generalization refers to the characteristic where performance can be guaranteed across various datasets. However, there are limitations to the generalization of AI / ML models. In other words, the performance of an AI / ML model can be guaranteed for data collected under a somewhat limited data collection configuration. Therefore, wireless communication networks can define a data collection configuration that guarantees the performance of AI / ML models and assign an Associated ID to each data collection configuration.
[0086] A wireless communication network can define a set (range) of generalizable datasets where the performance of AI / ML models is guaranteed, and determine a data collection configuration to correspond to this. The wireless communication network can define various data collection configurations and assign an Associated ID to each of them. The wireless communication network can train or develop AI / ML models according to the data collection configuration corresponding to each Associated ID.
[0087] The method for assigning Associated IDs may vary depending on the use case of the AI / ML model. For example, for AI / ML models used for beam management, data collection configurations may be determined based on the same downlink spatial domain transmission filter, reference signals in Quasi Co-location (QCL) status, or antenna panels used for transmission and reception. A different Associated ID may be assigned to each of the data collection configurations. In the example of CSI compression and prediction, data collection configurations may be determined based on SNR, CQI, PMI, RI representing channel environment status, and antenna panels used for transmission and reception. Associated IDs are uniquely defined within the network and can be classified into Global Associated IDs representing the same environment and Local Associated IDs valid only for certain cell groups / cells / sites / regions or equipment vendors. In the following, the Associated ID may correspond to at least one of the Global Associated ID and the Local Associated ID.
[0088] Each AI / ML model can be assigned a Model ID. An Associated ID can correspond to at least one Model ID. As another example, an Associated ID can correspond to multiple Model IDs. The Associated ID and Model ID can have a many-to-many correspondence.
[0089] For example, if the downlink spatial domain transmission filters are configured as a number (N), the Associated IDs for beam management can also be configured as a number (N), and the AI / ML models used for the corresponding AI / ML beam management can be configured as N corresponding. Similarly, if the base station (110-1) configures a number (N) of reference signal / resource sets that are not in a QCL state, the AI / ML models used for the corresponding AI / ML beam management and the associated IDs can be configured as N corresponding. On the other hand, if the AI / ML models have excellent generalization features and their performance is guaranteed across all datasets, they can be configured and operated with a small number (less than N) of AI / ML models.
[0090] In a 6G Massive MIMO environment, there is a Spatial Non-Stationarity (SNS) characteristic where channel statistics vary for each position or segment within the array. Therefore, in 6G networks, there is similarity in the wireless environment at the segment (= physical sub-array or virtual division) level, and Associated ID can be constructed based on this similarity. When the segment is single (whole array), there is no virtual division, and the time and frequency fields are fixed as the basic measurement units, the 6G Massive MIMO environment has a concept equivalent to the existing Associated ID.
[0091] A 6G-Associated ID is defined as a combination of a core spatial context, an SNS segment context, a time / frequency window, and aggregation rules. Sharing the same 6G-Associated ID implies that the data was processed under the same segment splitting, label, weighting, aggregator, and time / frequency window conditions. The core spatial context includes the same downlink spatial domain transmission filter, reference signal for QCL status, and transmit / receive antenna panels (including beamforming chains / modes if necessary), ensuring compatibility with 5G. The Spatial Non-Stationary segment context includes the definition and version of the segment splitting, the set of segment identifiers actually included, the version of the segment environment label generation algorithm, and weighting rules based on distance, path power, SNR, etc., and can add near / far distinctions to the concept as needed.
[0092] The 6G-Associated ID is also uniquely defined within the network and, as before, can be classified into a global associated ID and a local associated ID limited to specific cell groups, cells, sites, regions, or network manufacturers. The relationship with AI / ML models is also extended, allowing for a 1:1 or N:N (many-to-many) correspondence between the 6G-Associated ID and the model ID.
[0093] For example, in the case of beam management, if multiple downlink spatial domain transmission filters are configured, the array is divided into K segments, and a P90 aggregator is applied, the data collection configuration is identified in units of a fixed 6G-Associated ID that includes the filter, QCL, and panel, as well as the segment division and segment set, label algorithm, weighting rule, time / frequency window, and aggregator. In the case of CSI compression and prediction, since the segment label and weighting / aggregation rule affect the results in addition to the SNR, CQI, PMI, RI, and transmit / receive antenna panel, a 6G-Associated ID containing this context can function as an identification key for the data collection configuration.
[0094] In addition, when there is only one segment, no virtual partitioning, the aggregator is the Identity, and the time / frequency window is fixed as the basic measurement unit, the 6G-Associated ID is reduced to be virtually identical to the existing Associated ID, so it is naturally backward compatible with 5G-centric operations. In short, the 6G-AID can be described as an extended identifier that strictly guarantees reproducibility and equivalence judgment throughout the entire process of data collection, learning, inference, and verification by preserving the meaning of the associated identification used in 5G while adding the segment, window, and aggregation contexts necessary for SNS and PDV operations.
[0095] The operation of AI / ML models can be broadly classified into single-sided operation (one-side AI / ML model) and joint operation (two-side AI / ML model). In the case of a one-side AI / ML model, the AI / ML model is operated on only one side, either the base station or the terminal, whereas in the case of a two-side AI / ML model, the AI / ML model can be operated simultaneously on both the base station and the terminal.
[0096] Lifecycle Management (LCM) for AI / ML can be performed in various ways. First, the scope of LCM application can be determined by the AI / ML model ID. A base station or terminal can determine the scope of LCM application based on at least one AI / ML model ID. A base station or terminal can determine at least one AI / ML model to which LCM is applied based on at least one AI / ML model ID.
[0097] Secondly, the scope of application of AI / ML can be determined by a function ID or function identifier. Function-based LCM can be applied to AI / ML model(s) associated with at least one function without the need to precisely specify the AI / ML model between the base station and the terminal. The base station or the terminal can perform LCM-based operations for the AI / ML model(s) associated with a specific function.
[0098] Thirdly, the scope of application of AI / ML can be determined by an Associated ID. An Associated ID-based LCM can be applied to AI / ML model(s) corresponding to at least one Associated ID without the need to precisely specify the AI / ML model between the base station and the terminal. The base station or the terminal can perform LCM-based operations for the AI / ML model(s) corresponding to a specific Associated ID.
[0099]
[0100] Figure 6 is a diagram showing the prediction error according to the training of an AI / ML model.
[0101] In Figure 6, the horizontal axis represents the number of training cycles or epochs, and the vertical axis represents the prediction error of the AI / ML model. The L1 curve in Figure 6 represents the validation error of the AI / ML model, and the L2 curve represents the training error. Here, the training error represents the error between the value predicted by the AI / ML for the data used for training and the correct answer, and the validation error represents the error between the value predicted by the AI / ML for the validation data not used for training and the correct answer.
[0102] Referring to Figure 6, as the training cycle of an AI / ML model increases, the learning error tends to decrease, but the validation error tends to increase after a certain point. This phenomenon is called overfitting. In other words, if there are insufficient training cycles, underfitting occurs, and if there are too many training cycles, overfitting may occur.
[0103] In mobile communication environments, training data is mostly a partial dataset of environmental data acquired by communication nodes. If an AI / ML model is overfitted or underfitted to this training data, the reliability of the AI / ML model's inference results may decrease and performance may degrade.
[0104] To prevent such performance degradation, verification of AI / ML is necessary. During the verification process, monitoring and testing of AI / ML performance can be performed. Verification may be conducted in an actual wireless communication environment or in a test environment that simulates an actual wireless communication environment.
[0105] AI / ML validation may include post-deployment verification. Communication nodes may perform updates and / or fine-tuning of AI / ML models while operating AI / ML. The data used in the AI / ML training environment or conformance validation may differ from the data collected in the actual field (real-world communication environment). This may lead to data drift or data mismatch phenomena.
[0106] The communication node can perform fine-tuning or updates to the AI / ML model after the AI / ML model has been deployed. As another example, the communication node may change and / or switch the AI / ML model to a different AI / ML model than the existing one. In the cases described above, the communication node may perform a verification procedure after deployment.
[0107] The verification of AI / ML can be performed by a first communication node and a second communication node. In the following, the first communication node may include at least one of a base station and an emulator simulating a base station. The second communication node may include at least one of a terminal and an emulator simulating a terminal.
[0108] At least one of the first and second communication nodes (110, 120) can adjust a threshold value at which a certain function is triggered in the LCM in an AI / ML verification environment.
[0109] A threshold value may be used to trigger at least one of the following: a switch function of an AI / ML model (or feature, or model or training data associated with an associated ID), deactivation of AI / ML, fallback to Non-AI / ML, and AI / ML-related performance reporting function. For example, when the condition of the threshold value is satisfied, at least one of the following may be induced: a switch function of an AI / ML model (or feature, or model or training data associated with an associated ID), deactivation of AI / ML, fallback to Non-AI / ML, and AI / ML-related performance reporting function.
[0110] A threshold can represent the minimum performance standard for an AI / ML model. For example, if the performance metric of an AI / ML model falls below the threshold, the communication node may fall back to a legacy algorithm or deactivate the AI / ML.
[0111] Thresholds can serve as a criterion for determining pass or fail in conformance and post-deployment tests. For example, if the performance metrics of an AI / ML model fall below a threshold, the communication node may determine that the model has failed the verification.
[0112] The threshold value may be used to determine the conditions under which at least one of the first and second communication nodes (110, 120) performs a management request, a management instruction, and a performance report. When the conditions of the threshold value are satisfied, at least one of the first and second communication nodes (110, 120) may perform at least one of a management request, a management instruction, and a performance report.
[0113] At least one of the first communication node and the second communication node can separately set a threshold value used in an AI / ML verification environment and a threshold value used in an AI / ML general operation environment.
[0114] The following describes a method for setting threshold values used in an AI / ML verification environment. In the following, the first communication node may include at least one of a base station, gNB, network node, edge node, aggregator, or an emulator that simulates these. The second communication node may include at least one of a terminal, UE, edge node, AI / ML model execution entity, or an emulator that simulates these.
[0115] FIG. 7 is a flowchart illustrating an AI / ML verification method performed by a communication node according to an exemplary embodiment. In FIG. 7, the first communication node (110) may be referred to as a network, distinct from the terminal.
[0116] Referring to FIG. 7, in step S110, the first communication node (110) can transmit a terminal function request (UECapabilityEnquiry) message. The second communication node (120) can receive the terminal function request message. The second communication node (120) can collect information on supported AI / ML functions. The second communication node (120) can determine supported AI / ML functions by considering the hardware performance of the second communication node (120), the status of the AI / ML model, and whether inference support is possible. The AI / ML functions supported by the second communication node (120) can be indicated by at least one of an AI / ML model ID, an AI / ML function ID, and an associated ID. The second communication node (120) can configure terminal function information based on the information on supported AI / ML functions.
[0117] In step S120, the second communication node (120) can transmit a terminal capability information (UECapabilityInformation) message. The first communication node (110) can receive the terminal capability information message.
[0118] The first communication node (110) can generate first configuration information based on supported AI / ML function information included in terminal function information. The first configuration information may also be referred to as first RRC configuration information or first network configuration information.
[0119] The first configuration information may include network-side configuration information for implementing AI / ML functions that the second communication node (120) can support. For example, the first configuration information may include at least one of the following: information regarding AI / ML functions allowed by the network side among the AI / ML functions that the second communication node (120) can support, and a configuration regarding a message that the second communication node (120) must report to the network regarding the use of AI / ML functions.
[0120] In step S130, the second communication node (120) can transmit the first configuration information. The second communication node (120) can obtain the first configuration information.
[0121] In step S140, the second communication node (120) can determine applicable functionality based on the first configuration information. Here, 'applicable functionality' may include AI / ML functions available for communication between the first communication node (110) and the second communication node (120).
[0122] The second communication node (120) can determine applicable functions by considering information regarding AI / ML functions allowed by the network side and information regarding AI / ML functions that the second communication node (120) can support. The second communication node (120) can set a method for transmitting reporting messages to the first communication node (110) or the network while operating AI / ML functions based on the first configuration information.
[0123] In step S150, the second communication node (120) can transmit an applicable functionality report based on the applicable AI / ML functionality determined in step S140. For example, the second communication node (120) can transmit the applicable functionality report by including it in a UE Assistance message. As another example, the second communication node (120) can transmit the applicable functionality report through a message separate from the UE Assistance message. The first communication node (110) can receive the AI / ML functionality report.
[0124] In step S160, the first communication node (110) may transmit second configuration information. The second configuration information may also be referred to as first RRC configuration information or second network configuration information.
[0125] The first communication node (110) may include an inference configuration in the second configuration information. The first communication node (110) may transmit an inference configuration based on an applicable function report. The first communication node (110) may transmit an inference configuration required when using an AI / ML model corresponding to the applicable function. The second communication node (120) may obtain the inference configuration by receiving the second configuration information. The second communication node (120) may determine an inference method using AI / ML based on the inference configuration.
[0126] The above description explains the case where the inference configuration is included in the second configuration information. However, the embodiments are not limited thereto. For example, the inference configuration may be included in the first configuration information. In this case, the second communication node (120) may specifically determine an inference method using an AI / ML model corresponding to a function applicable in step S140. If the inference configuration is included in the first configuration information, step S160 may be omitted. However, even if the inference configuration is included in the first configuration information, if an update to the inference configuration occurs after step S150 is performed, the first communication node (110) may transmit the second configuration information.
[0127] The first communication node (110) can generate AI / ML configuration information based on information regarding the threshold value described above. The first communication node (110) can include AI / ML configuration information in the first configuration information transmitted at step S130. The threshold value included in the AI / ML configuration information can be used to trigger at least one of the switching function of the AI / ML model (or function, or model or training data associated with the associated ID), the deactivation of AI / ML, and the fallback function to Non-AI / ML, as described above.
[0128] A threshold can represent the minimum performance standard for an AI / ML model. For example, if the performance metric of an AI / ML model falls below the threshold, the communication node may fall back to a legacy algorithm other than AI / ML or deactivate the AI / ML. Therefore, the lower the threshold is set, the less frequently fallback, deactivation, switch, and report actions may occur. The higher the threshold is set, the more frequently fallback, deactivation, switch, and report actions may occur. However, the embodiments are not limited to this. Depending on the type of metric to which the threshold is applied, the manner in which the frequency of fallback, deactivation, switch, and report actions changes with changes in the threshold may vary. For example, if a smaller value of a specific metric indicates a higher performance standard, setting a lower threshold for that metric may increase the frequency of fallback, deactivation, and switch actions.
[0129] The first communication node (110) can set different threshold values for the normal operation state of AI / ML and the verification state of AI / ML. For example, the first communication node (110) can apply a first threshold value in the normal operation state of AI / ML and apply a second threshold value in the verification state of AI / ML.
[0130] For example, the second threshold value may represent a performance standard lower than the first threshold value. For example, the second threshold value may be set smaller than the first threshold value.
[0131] The first communication node (110) can improve the efficiency and effectiveness of the AI / ML verification process by setting the second threshold value differently from the first threshold value. For example, by setting the second threshold value lower than the first threshold value, the intervention of stochastic variation effects occurring during the AI / ML verification process can be mitigated. AI / ML inference performance can fluctuate stochastically depending on the wireless environment, traffic conditions, etc. By applying a mitigated (low) threshold value during the AI / ML verification process, it is possible to prevent the verification from being interrupted or the conclusion from being distorted due to temporary performance degradation in the short term. It is possible to prevent premature return, deactivation, and switching actions during the AI / ML verification process. By reducing the frequency of return, deactivation, and switching actions, statistically significant verification of the average performance and stability of the AI / ML can be performed.
[0132] As another example, the second threshold value may represent a higher performance standard than the first threshold value. For instance, the second threshold value may be set higher than the first threshold value.
[0133] The second communication node (120) can perform fallback, deactivation, switch, and report operations when the AI / ML inference performance is higher than the second threshold or the first threshold. By setting the second threshold higher than the first threshold, the effects of probabilistic performance fluctuations occurring during the AI / ML verification process can be observed more efficiently. AI / ML inference performance may fluctuate probabilistically depending on the wireless environment, traffic conditions, etc. A second threshold with a higher value (indicating higher performance) may be applied during the AI / ML verification process. When the AI / ML performance is above the second threshold, the second communication node (120) performs at least one of fallback, deactivation, switch, or report operations to observe the performance range of the AI / ML model (function) under verification that is superior to the existing first threshold, thereby enabling efficient verification by identifying rapid performance improvement or delivering fewer report messages.
[0134] By setting the second threshold value differently from the first threshold value, the first communication node (110) and / or the second communication node (120) can acquire sufficient data during the AI / ML verification process and sufficiently capture under- / over-fit signals to utilize for fine-tuning and data augmentation decision-making. Additionally, the second threshold value different from the first threshold value can be used to omit unnecessary return, disable, switch-over actions, and unnecessary performance reporting and control procedures. This reduces signaling overhead by lowering the frequency of AI / ML-related reporting and control actions that occur during the AI / ML verification process.
[0135] By setting the second threshold lower than the first threshold, false positive results can be suppressed during the AI / ML verification process. Conversely, during normal operation, the first threshold is applied to reduce the risk of neglecting performance degradation and enable timely actions such as reverting, deactivating, and switching.
[0136] The first communication node (110) may include the first threshold value and the second threshold value in the first configuration information transmitted in step S130. If the first threshold value is shared in advance between the first communication node (110) and the second communication node (120) through another message, the first communication node (110) may include the second threshold value in the first configuration information.
[0137] The AI / ML configuration information included in the first configuration information may include identification information of an AI / ML model to which a first threshold and a second threshold are applied. The identification information of the AI / ML model may include at least one of an AI / ML model ID, an AI / ML function ID, and an associated ID. When identifying a model to be verified by unit of AI / ML model ID, the target of identification can be subdivided and referred to. When identifying a model to be verified by unit of AI / ML function, AI / ML models associated with a specific function can be efficiently indicated and identified. When identifying a model to be verified by an associated ID, it may be easier to verify whether the training environment and the inference environment correspond in substance or are within a generalizable range. Through this, the verification subject can verify whether the AI / ML model is trained using a dataset that guarantees expected performance. If the first configuration information does not include a first threshold but includes a second threshold, the first configuration information may include identification information of an AI / ML model to which the second threshold is applied.
[0138] If the first configuration information includes a first threshold and / or a second threshold, the second communication node (120) may utilize the threshold set from the beginning when evaluating applicable functions in step S140. Additionally, in step S150, the second communication node (120) may generate and transmit applicable function reports by reflecting the preset threshold. That is, by presenting clear criteria from the beginning, consistency in AI / ML-related operations can be ensured and signaling overhead can be reduced.
[0139] The first communication node (110) may include AI / ML configuration information containing a first threshold and a second threshold in the second configuration information transmitted in step S160. If the first threshold is shared in advance between the first communication node (110) and the second communication node (120) through another message, the first communication node (110) may include AI / ML configuration information containing the second threshold in the second configuration information.
[0140] If the second configuration information includes a first threshold and / or a second threshold, in step S140, the second communication node (120) can determine applicable functions based on internal logic set by itself. Since the second configuration information is transmitted after steps S140 and S150 are performed, the first threshold and / or the second threshold can be set to reflect the final inference configuration and performance criteria. Thus, the first threshold and / or the second threshold can be set precisely and adaptively. The first threshold and / or the second threshold can be set to be more suitable for the actual communication environment.
[0141] The above explanation was based on the premise that the same metrics defined in both the AI / ML validation environment and the AI / ML operational environment are used. While the same metrics are used in both environments, the threshold values applied to those metrics may be set differently in the two environments.
[0142] Metrics representing the performance of AI / ML may be defined differently in an AI / ML verification environment and an AI / ML normal operation environment. That is, the metrics used in the AI / ML verification environment and the metrics used in the AI / ML normal operation environment may be different. The first communication node (110) and / or the second communication node (120) may use distinct metrics in the AI / ML verification process and the AI / ML normal operation process, respectively. Since the purpose and method of performance monitoring in the AI / ML verification and AI / ML operation processes are different, using distinct metrics allows each procedure to be performed more efficiently.
[0143] AI / ML validation can be performed before an AI / ML model is deployed. AI / ML validation can also be performed after deployment when fine-tuning or model updates occur. AI / ML validation can be performed for the purpose of verifying the effectiveness and performance of the AI / ML model.
[0144] In the operation of AI / ML, performance monitoring is a continuous procedure performed in a real network environment and can be used to make decisions such as fallback, switch, retraining, and model adaptation.
[0145] Using the same metrics for both AI / ML verification and performance monitoring during operation can cause network congestion and signaling overhead, thereby reducing overall system efficiency. In particular, during post-deployment testing where AI / ML performance is being verified, the need for endpoints to participate in all aspects of LCM decision-making, such as rollback and changes, may be reduced.
[0146] As mentioned above, if the metrics and thresholds used in AI / ML operations are utilized as is in the AI / ML validation process, the intervention of stochastic performance variability effects becomes severe. Furthermore, actions such as rollbacks or modifications may be performed before the AI / ML model is sufficiently validated, making it difficult to fully verify the effects of fine-tuning or updates. Additionally, unnecessary signaling overhead may occur.
[0147] For example, the first communication node (110) and / or the second communication node (120) may use a first metric that is relatively lightweight and aggregated in the AI / ML normal operation procedure (performance monitoring procedure during AI / ML operation). The first metric used in the AI / ML normal operation procedure may have relatively low computational complexity. Therefore, the first communication node (110) and / or the second communication node (120) may use relatively few computational resources to calculate the first metric. The first metric may be expressed with a relatively small amount of data. For example, the first metric may include a confidence score, output deviation, trigger count, etc., but the embodiments are not limited thereto.
[0148] For example, the first communication node (110) and / or the second communication node (120) may use a relatively fine-grained second metric in the AI / ML verification procedure. The second metric used in the AI / ML verification procedure may have relatively high computational complexity. The second metric may be represented by a relatively large amount of data. Therefore, the first communication node (110) and / or the second communication node (120) may utilize relatively large computational resources to calculate the second metric. For example, the second metric may include inference accuracy, latency, loss function values, etc., but the embodiments are not limited thereto.
[0149] The first communication node (110) and / or the second communication node (120) can reduce signaling overhead by using a first metric that is relatively intensive and simplified in the AI / ML operation procedure. The first communication node (110) and / or the second communication node (120) can save computational resources used to calculate the first metric. Since the calculation and reporting of the first metric may occur relatively frequently in the AI / ML operation procedure, network efficiency can be improved by reducing signaling overhead and computational resource consumption through the use of a first metric that is relatively intensive and simplified.
[0150] In the AI / ML operation procedure, the frequency of generating and reporting the second metric may be relatively low. The first communication node (110) and / or the second communication node (120) can accurately evaluate the actual effect of model updates or fine-tuning by using a relatively precise and fine-grained second metric in the AI / ML verification procedure.
[0151] Although not shown in FIG. 7, the second communication node (120) can perform AI / ML verification based on the AI / ML configuration information obtained in step S130 or S160 of FIG. 7.
[0152] Figure 8 is a diagram showing the configuration method of IE included in AI / ML configuration information.
[0153] Referring to FIG. 8, the first configuration information may include AI / ML configuration information. The AI / ML configuration information is shown in R10.
[0154] The parameters shown in Fig. 8 are explained through Table 2.
[0155] Parameter Name Description validationMode Indicates whether AI / ML validation mode is being performed (e.g.) TRUE -> The terminal is performing AI / ML validation FALSE -> The terminal is performing normal AI / ML operations) fallbackSuppressionFlag Controls whether to suppress fallback or model switch operations triggered by LCM during the AI / ML validation phase (e.g.) TRUE -> Suppress fallback / switch during AI / ML validation FALSE -> Allow fallback / switch validationPerformanceMetric Specifies the secondary metric used in the AI / ML validation procedure (enumerated option) inferenceAccuracy: Accuracy of inference results (e.g., correct prediction vs. full prediction) latency: Time taken to complete one inference cycle lossFunction: Error calculated during validation (cross-entropy or MSE) otherMetric: Additional metrics not explicitly listed runtimeMonitoringMetric Specifies the primary metric used to monitor AI / ML model performance during normal operation (enumerated option) confidenceScore: Average model confidence of predictions Level (e.g., softmax output).outputDeviation: Deviation of the model output compared to the expected behavior.triggerCount: The number of times a monitoring event or specific metric has exceeded a threshold (e.g., fallback trigger).other: Placeholder for future or custom metrics.reportingThreshold: Defines a threshold (percentage or score) that determines when the terminal must report the first or second metric. The value of reportingThreshold depends on the metric selected in the first or second metric.fallbackTriggerThreshold: Specifies the threshold at which a fallback is triggered. The value of fallbackTriggerThreshold depends on the metric selected in the first or second metric.
[0156] Referring to FIG. 8 and Table 2, the AI / ML configuration information may include information regarding which metric each of the first metric and the second metric is defined as. The AI / ML configuration information may include a threshold at which a return is triggered and a threshold at which a report is triggered. In addition to those shown in FIG. 8 and Table 2, other thresholds may be further included in the configuration information for AI / ML verification. For example, the configuration information for AI / ML verification may include a threshold at which a switch is triggered or a threshold at which AI / ML deactivation is triggered. The AI / ML configuration information may further include at least one of information regarding the time point and time interval at which AI / ML verification is performed. The second communication node (120) may determine the time point for performing AI / ML verification based on at least one of the information regarding the time point and time interval at which AI / ML verification is performed, which is included in the AI / ML configuration information. The second threshold, representing the AI / ML performance criterion, may be determined by comparing it with a measurement value reported in the form of a performance indicator (metric).
[0157] For example, in a Beam Management use case, the second communication node (120) reports a metric representing the number or ratio of accurately predicted reference signals within a predetermined window interval, and the network can use the number or ratio of accurately predicted reference signals to verify the performance of the AI / ML model by comparing it with a second threshold value. That is, the second metric may represent the number or ratio of reference signals that the second communication node (120) has predicted with an accuracy greater than a predetermined standard using the AI / ML model, and the second communication node (120) can verify the performance of the AI / ML model based on the comparison result between the second metric and the second threshold value.
[0158] The number of reference signals that the second communication node (120) predicts with an accuracy greater than a predetermined standard using an AI / ML model can correspond to the number of matches between the Top-M beams configured by the network (top M beams extracted based on L1-RSRP) and the Top-K beams (K beams predicted by the AI / ML model as having excellent reception performance) that the second communication node (120) predicts have excellent reception performance.
[0159] The ratio of reference signals predicted by the second communication node (120) using an AI / ML model with an accuracy greater than a predetermined standard for reception performance can be determined based on the ratio of overlap with the Top-M beam among the Top-K beams.
[0160] Accordingly, in a beam management use case, a measurement value (second metric) representing the number of reference signals (beams) predicted with an accuracy greater than a predetermined standard by the second communication node (120) may correspond to one example of an AI / ML performance indicator (measure value, metric) included in the comparison range of the second threshold standard of the present invention. Additionally, the network (i.e., the first communication node (110)) may configure multiple beams to improve the precision of verification (configuring multiple beams that the terminal must report), and the second communication node (120) may calculate the number of reference signals or the ratio value accurately predicted for each of the multiple beams by sharing and reusing the same measurement (L1-RSRP) and the same inference (Top-K) results. The second communication node (120) may package the composition of the measurement value (indicator, metric) reported for verification into a single synthetic report message and transmit it, or bundle multiple report compositions into a single transmission and report them. The second communication node (120) can report to the first communication node (110) the inference results for multiple reference signals (multiple beams) in a single report during the process of using an AI / ML model (function) based on AI / ML configuration information. Accordingly, multiple performance indicators are provided without additional measurement and inference, thereby reducing the overhead of message exchange and control signals required for reporting, as well as the computational load of the second communication node (120).
[0161] Figure 9 is a diagram showing an example of the implementation of AI / ML configuration information.
[0162] Referring to FIG. 9, since the validationMode in the first area (R22) is set to FALSE, the AI / ML configuration information described in the first area (R22) can be utilized when the AI / ML is operating normally. Since the validationMode in the second area (R32) is set to TRUE, the AI / ML configuration information described in the second area (R32) can be utilized in AI / ML verification.
[0163] The AI / ML configuration information described in the first area (R22) may include information regarding the first metric. For example, the first metric (runtimeMonitoringMetric) may be designated as the confidence score. The AI / ML configuration information described in the second area (R22) may include information regarding the second metric. For example, the second metric (validationPerformanceMetric) may be designated as inference accuracy.
[0164] The AI / ML configuration information described in the first area (R22) may include at least one threshold value related to the first metric. The AI / ML configuration information described in the second area (R32) may include at least one threshold value related to the second metric. The threshold value related to the first metric and the threshold value related to the second metric may each be set independently and differently from each other.
[0165] FIG. 10 is a diagram showing the configuration method of IE included in AI / ML configuration information. In describing the embodiment of FIG. 10, content that overlaps with FIG. 8 is omitted.
[0166] Referring to FIG. 10, the AI / ML configuration information may further include reportingLogic representing AI / ML-related reporting conditions. For example, reportingLogic may represent a comparison direction used to trigger the second communication node (120) to make AI / ML-related reports. That is, reportingLogic may represent signaling logic for AI / ML-related reports. When reportingLogic is set to reportAbove, the second communication node (120) may make an AI / ML-related report when a first metric or a second metric representing the performance of AI / ML is greater than a threshold value. When reportingLogic is set to reportBelow, the second communication node (120) may make an AI / ML-related report when a first metric or a second metric representing the performance of AI / ML is less than a threshold value.
[0167] FIG. 11 is a diagram showing an example of the implementation of AI / ML configuration information. In describing the embodiment of FIG. 11, content that overlaps with FIG. 9 is omitted.
[0168] Referring to FIG. 11, since the validationMode in the first area (R24) is set to FALSE, the AI / ML configuration information described in the first area (R24) can be utilized when the AI / ML is operating normally. Since the validationMode in the second area (R34) is set to TRUE, the AI / ML configuration information described in the second area (R34) can be utilized in AI / ML verification.
[0169] In the AI / ML configuration information described in the first area (R24), the reportingLogic can be set to reportAbove. Accordingly, when the AI / ML is operating normally, the second communication node (120) can perform AI / ML-related reporting when the confidenceScore is greater than the reportingThreshold of 85. The second communication node (120) can notify the first communication node (110) that a performance reporting triggering condition has occurred through the AI / ML-related reporting. The second communication node (120) can notify the first communication node (110) of information regarding at least one of the status of the AI / ML model and the performance of the AI / ML model through the AI / ML-related reporting.
[0170] In the AI / ML configuration information described in the second area (R34), the reportingLogic can be set to reportBelow. Accordingly, when the AI / ML is operating normally, the second communication node (120) can perform AI / ML-related reporting when the inference accuracy is less than the reporting threshold of 90.
[0171] As described above, the flexibility of communication network operation can be increased by using a parameter indicating a comparison direction used to trigger the second communication node (120) to make an AI / ML-related report.
[0172] For example, since the second metric used in the AI / ML verification process and the first metric used in the normal operation process of AI / ML are different, the reporting logic for each of the first and second metrics may be different. Therefore, the first communication node (110) can clearly control the reporting direction and reporting conditions by adding a reportingLogic field to the AI / ML configuration information.
[0173] The above describes the case where AI / ML configuration information is included in the first configuration information transmitted in step S130 of FIG. 7. However, the embodiments are not limited thereto. As with the description of the threshold value above, the AI / ML configuration information may also be included in the second configuration information.
[0174] After the deployment of the AI / ML model is completed, the AI / ML model may be updated (fine-tuned or updated) by the second communication node (120). Under current communication standards, there is no procedure to explicitly signal to the first communication node (110) even if the second communication node (120) updates the AI / ML model on its own. If the second communication node (120) updates the AI / ML model on its own and does not notify the first communication node (110), the network-side first communication node (110) may not be able to track or verify the updated model status. Furthermore, if the second communication node (120) assigns a new AI / ML model ID to the updated AI / ML model according to a predetermined rule or a rule it has set itself during the update process, information regarding the AI / ML model ID is not shared between the first communication node (110) and the second communication node (120), which may cause problems in future LCM full-cycle management.
[0175] According to the embodiment described below, the second communication node (120) can signal information about the update to the first communication node (110) after updating the AI / ML model. In particular, when post-deployment verification is performed network-driven, the signaling of the AI / ML model update can cause the first communication node (110) to perform post-deployment verification. Additionally, the signaling of the AI / ML model update can facilitate full-cycle management of the LCM by allowing update information regarding the AI / ML model to be shared between the first communication node (110) and the second communication node (120).
[0176] Figure 12 is a flowchart illustrating an AI / ML verification method that includes a process of transmitting a signaling indicating an AI / ML model update.
[0177] Referring to FIG. 12, in step S210, the second communication node (120) can perform an update of the AI / ML model. For example, the second communication node (120) can perform additional training or fine-tuning of the AI / ML model locally. For example, the second communication node (120) can perform an update of the AI / ML model based on the network, i.e., the association ID set by the first communication node (110). The second communication node (120) can perform an update of the AI / ML model by configuring training data or configuring a training environment with conditions corresponding to the pre-set association ID. The second communication node (120) can incrementally update the AI / ML model. The second communication node (120) can update the model by gradually adjusting or fine-tuning the existing AI / ML model parameters.
[0178] In step S220, the second communication node (120) may generate an AI / ML update report. The AI / ML update report may include identification information of the updated AI / ML model. The AI / ML update report may include at least one of an AI / ML model ID, an AI / ML function ID, and an association ID corresponding to the updated AI / ML model. The AI / ML update report may include a field (modelUpdateTimeStamp) regarding the time when the AI / ML model was updated. The AI / ML update report may include a field (modelUpdateReason) regarding the reason why the AI / ML model was updated.
[0179] The field regarding the time when the AI / ML model is updated may include more detailed information. For example, the field regarding the time when the AI / ML model is updated may include information regarding at least one of the time when the second communication node (120) updates the AI / ML model (function) and the time when the updated AI / ML model (function) is applied in the field by the second communication node (120).
[0180] In step S230, the second communication node (120) can transmit an AI / ML update report. The first communication node (110) can receive the AI / ML update report. In step S240, the first communication node (110) can transmit an ACK message indicating that it has received the AI / ML update report. The second communication node (120) can receive the ACK message.
[0181] The first communication node (110) can identify an updated AI / ML model based on the identification information of the AI / ML model included in the AI / ML update report. The first communication node (110) can identify an updated AI / ML model based on at least one of an AI / ML model ID, an AI / ML function ID, and an association ID. The first communication node (110) can check the data collection history or data change details corresponding to the association ID based on the association ID. The first communication node (110) can determine the AI / ML verification schedule by confirming the time when the AI / ML model was updated. The first communication node (110) can determine an appropriate AI / ML verification method and / or LCM full-cycle management method by confirming and analyzing the cause of the update. After synchronizing information regarding the AI / ML model management records with the second communication node (120), the first communication node (110) can perform the procedures necessary for AI / ML verification and LCM full-cycle management.
[0182] After the second communication node (120) updates the AI / ML model (function), the updated AI / ML model (function) may be applied based on the second communication node's (120) own judgment. In this case, as described above, the AI / ML update report may include information about the time when the updated AI / ML model (function) was applied.
[0183] As another example, after the second communication node (120) updates the AI / ML model (function), the updated AI / ML model (function) can be applied based on the judgment of the first communication node (110). That is, the first communication node (110) can determine the timing of the application of the updated AI / ML model (function). In this case, the AI / ML update report may not include information regarding the timing of the application of the updated AI / ML model (function). After step S230, the first communication node (110) can transmit information regarding the timing of the application of the updated AI / ML model (function) to the second communication node (120) through a separate signaling procedure.
[0184] In the previous explanation, the AI / ML validation methods used for beam management were described based on a comparison of Top-M and Top-K beam sets. Below, the AI / ML validation methods used for beam management are explained in more detail.
[0185] Conventional methods for verifying AI / ML-based prediction or judgment results often relied on comparing one or more metric values derived from the output of an AI / ML model with a pre-set threshold and determining the performance or usability of the AI / ML model based on whether the metric value satisfies the threshold.
[0186] However, this threshold-based verification method has limitations in that the appropriate threshold value may vary depending on the wireless environment, the mobile state of the terminal, temporal variations in beam quality, differences between training data and the actual operating environment, and changes in network configuration information. For example, in a specific environment, the AI / ML-based beam prediction results may be evaluated as sufficiently valid even with the same metric value, whereas in another environment, the same metric value may imply unreliable results. Therefore, a method that simply compares a single metric value with a fixed threshold may not adequately reflect the actual performance, stability, or applicability of AI / ML-based operations.
[0187] Accordingly, the AI / ML verification method according to one embodiment is not limited to a method of determining pass / fail by simply comparing a metric value derived from an AI / ML inference result with a preset threshold. The verification method according to one embodiment of the present invention can verify AI / ML-based beam management performance based on at least one of a plurality of measurement criteria, a set of measurement criteria beams, a set of prediction candidate beams, a correspondence between a beam result and an AI / ML inference result, a change pattern per time instance, or a relative relationship between multiple metrics.
[0188] That is, the verification method according to one embodiment may be a verification method that comprehensively considers structural consistency between measurement results and AI / ML inference results, the degree of overlap between candidate sets, the ranking or relative suitability of prediction results, temporal consistency, or patterns of performance change according to configuration information, rather than simply whether a fixed threshold is satisfied. Accordingly, the present invention can more flexibly and reliably determine whether AI / ML-based beam management operations can be effectively used in an actual wireless environment without relying excessively on a specific threshold setting.
[0189] According to the AI / ML verification method described below, the performance of AI / ML-based beam management can be verified by utilizing the relationship between the reference beam set and the prediction candidate beam set, verification results for each number of multiple references, and / or verification results from multiple time instances. This AI / ML verification method can be adaptedly applied to various wireless environments and configuration conditions, more so than a simple judgment method based on a fixed threshold, and enables the maintenance, limited application, fallback behavior, or reconstruction of verification configuration information of AI / ML-based beam management based on the verification results.
[0190] FIG. 13 is a flowchart illustrating an AI / ML verification method performed by a second communication node (120) according to an exemplary embodiment.
[0191] The AI / ML verification method of FIG. 13 is not limited to a method of simply comparing a single metric value corresponding to an AI / ML inference result with a preset threshold, but may be a method of verifying AI / ML-based beam management performance based on at least one of the inclusion relationship, overlap relationship, intersection relationship, and membership relationship between a set of measurement reference beams derived based on measurement results and a set of prediction candidate beams derived based on AI / ML inference results.
[0192] Referring to FIG. 13, in step S1310, the second communication node (120) can obtain AI / ML verification configuration information from the first communication node (110). Here, the AI / ML verification configuration information may refer to configuration information used for verifying an AI / ML-based beam management function, an AI / ML model, an AI / ML inference function, or an AI / ML-based prediction function.
[0193] AI / ML validation configuration information may include information regarding multiple measurement count M values. For example, multiple measurement count M values may be set as {1, 2, 4}. However, it is not limited thereto, and the AI / ML validation configuration information may also include a single measurement count M value. The measurement count M may refer to the number of higher-level measurement beams selected to form a set of measurement beams from beam measurement results. Multiple M values may be values for obtaining validation results of different strictness levels.
[0194] In one embodiment, one or more M values included in the AI / ML validation configuration information may be information for determining the granularity of the AI / ML validation. Here, the M values may represent the number of measurement reference beams selected from the beam measurement results, and the level of granularity of the validation results may vary depending on the number of M values, range, intervals between M values, etc.
[0195] The AI / ML validation configuration information may include information regarding the number of prediction candidates K, or it may not explicitly include information regarding the number of prediction candidates K. The number of prediction candidates K may refer to the number of top prediction beams selected to form a set of prediction candidate beams from the AI / ML inference results. In one embodiment, and in another embodiment, the value of K may be determined by the AI / ML validation configuration information, the output structure of the AI / ML model, or a rule agreed upon in advance between the first communication node (110) and the second communication node (120).
[0196] For example, if the AI / ML verification configuration information includes only one M value, the second communication node (120) can produce one verification result corresponding to that M value. On the other hand, if the AI / ML verification configuration information includes multiple M values, the second communication node (120) can evaluate the AI / ML-based beam prediction performance at multiple levels of granularity by producing a verification result for each M value.
[0197] For example, M=1 can correspond to a relatively strict validation granularity to verify whether the AI / ML model's predicted candidate beam set includes the top measurement beam based on actual measurement results, while larger M values, such as M=2, M=4, or M=8, can correspond to a relatively relaxed validation granularity to verify whether the AI / ML model's predicted candidate beam set captures a candidate beam range with excellent actual measurement quality. Accordingly, validation results for multiple M values can be used as a performance profile indicating at which level of measurement reference beam range the AI / ML model's predicted candidate beam set is valid.
[0198] In one embodiment, the AI / ML validation configuration information may further include a value K, which is the number of prediction candidates. In another embodiment, the value K is not explicitly included in the AI / ML validation configuration information and may be a value pre-set in the second communication node (120), a value determined by the output structure of the AI / ML model, or a value pre-agreed upon between the first communication node (110) and the second communication node (120). In another embodiment, the second communication node (120) may independently determine the value K corresponding to the value M. For example, the second communication node (120) may set the value K to be equal to the value M, or calculate the value K based on a pre-set function based on the value M.
[0199] In one embodiment, the number of prediction candidates K may be determined by the number of prediction beams included in the AI / ML inference result report per time instance. For example, if the AI / ML inference result report for a specific time instance includes K prediction beams, the second communication node (120) may derive a set of prediction candidate beams including the K prediction beams.
[0200] In addition to the above, the AI / ML verification configuration information may further include at least one of the following: information for identifying the AI / ML model or AI / ML function to be verified; information for identifying the beam management function to be verified; information regarding the verification time or verification interval; information regarding monitoring resources; information regarding the reference signal to be measured; information regarding the type of measurement result; information regarding the reporting period or reporting conditions of the verification report; information regarding the format of the verification report; information regarding the calculation method of verification results by M value; information regarding the reference value or threshold value; and information regarding the transmission trigger conditions of the verification report. For example, the monitoring resources may include CSI-RS, SSB, or other reference signal resources, and the measurement results may include at least one of L1-RSRP, RSRQ, SINR, or beam quality indicators.
[0201] In step S1320, the second communication node (120) measures a beam at least one time instance and can derive a set of measurement reference beams (Top-M beam set) corresponding to the number of measurement references M based on the beam measurement results. The set of measurement reference beams may include the top M beams with good measurement quality in the corresponding time instance. For example, the second communication node (120) may sort multiple beams measured in a specific time instance according to L1-RSRP or a beam quality indicator and derive the top M beams among them as the set of measurement reference beams. Here, the time instance may be a validation occasion, a measurement time, a reporting time, or a time unit in which the AI / ML inference result and the measurement result are related.
[0202] For example, the second communication node (120) may measure at least one of L1-RSRP, RSRQ, SINR, received power, channel quality, and beam quality for CSI-RS, SSB, or other reference signal resources, and derive a Top-M beam set including the top M beams based on the ranking of the measurements. The Top-M beam set may include at least one of M beam identifiers, beam indices, and reference signal resource identifiers that are determined to be the best by the measurement results at a specific time instance.
[0203] When the number of measurement criteria M is set to a single value, a single set of measurement criteria beams can be derived for the corresponding time instance. On the other hand, when the number of measurement criteria M is set to multiple values, the second communication node (120) can derive multiple sets of measurement criteria beams by applying different number of measurement criteria M for each value of M.
[0204] In step S1330, the second communication node (120) may derive a set of predicted candidate beams (Top-K beam set) based on AI / ML inference results at least one time instance. The set of predicted candidate beams may include the top K predicted beams among the beams predicted by the inference results of the AI / ML model. For example, if the AI / ML inference results include priority or ranking information of beam indices, the second communication node (120) may derive a set of predicted candidate beams by selecting the top K beams from the ranking information. The AI / ML inference results may include beam prediction results output by the AI / ML model, beam ranking information, a list of candidate beams, a score per beam, a confidence value per beam, or a combination thereof.
[0205] If the number of prediction candidates K is fixed regardless of the number of measurement criteria M, a single set of prediction candidate beams can be derived for the corresponding time instance. On the other hand, if the number of prediction candidates K is set for each value of M depending on the number of measurement criteria M, the second communication node (120) may derive multiple sets of prediction candidate beams by applying different number of prediction candidates K for each value of M.
[0206] Steps S1320 and S1330 are not limited to the order shown in FIG. 13. For example, the second communication node (120) may first derive a Top-K beam set based on AI / ML inference results and then derive a Top-M beam set based on measurement results, or the two steps may be performed in parallel or at least partially overlapping.
[0207] Additionally, when deriving Top-M beam sets and Top-K beam sets for multiple M values, the second communication node (120) can reuse at least one of the measurement results performed once and the AI / ML inference results performed once. For example, the second communication node (120) can derive a measurement reference beam set corresponding to multiple M values from the same measurement result. Additionally, the second communication node (120) can derive a prediction candidate beam set corresponding to K values from the same AI / ML inference result and use it commonly for calculating verification results for multiple M values. Accordingly, multiple verification results can be produced without repeatedly performing separate beam measurements or separate AI / ML inference for each M value.
[0208] In step S1340, the second communication node (120) can output a verification result for each M value. The verification result can be calculated based on the relationship between the Top-M beam set and the Top-K beam set corresponding to a specific M value.
[0209] The verification result for each M value can be calculated based on at least one of the intersection, inclusion relationship, overlap relationship, and the number of elements of the intersection between the measurement standard beam set (Top-M beam set) corresponding to the number of measurement standards M and the prediction candidate beam set (Top-K beam set) corresponding to the number of prediction candidates K.
[0210] For example, the validation result for each value of M can be calculated based on whether the intersection of the set of measurement beams corresponding to the number of measurement criteria M at a specific time instance and the set of prediction candidate beams is an empty set.
[0211] For example, the second communication node (120) can determine the verification result for each M value based on mathematical formula 1.
[0212]
[0213]
[0214] In mathematical formula 1 represents the set of measurement beams (Top-M beam set) derived at time instance t, and represents the set of prediction candidate beams (Top-K beam set) derived at time instance t. Referring to Equation 1, The result value may vary depending on whether the intersection between the Top-M beam set and the Top-K beam set is an empty set. For example, the second communication node (120) is If so, determine the verification result for the corresponding M value as successful, and If so, the verification result for the corresponding M value may be determined as a failure.
[0215] According to the present embodiment, it is possible to simply determine whether at least one of the Top-K candidate beams predicted by the AI / ML model is included in the Top-M beam with excellent actual measurement quality. Accordingly, it is possible to quickly verify whether the AI / ML-based beam prediction provided valid candidate beams without complex numerical calculations. In addition, since only the existence of an intersection is utilized, the verification result can be expressed as 1 bit or a small number of indicators, thereby reducing the size of the verification report and enabling efficient AI / ML verification even on low-complexity terminals.
[0216] In another embodiment, the verification result for each M value may be calculated based on whether the number of elements in the intersection of the prediction candidate beam set and the metric beam set at a specific time instance is greater than or equal to a threshold value. The threshold value may be predetermined by AI / ML verification configuration information or may be determined differently depending on the M value. The second communication node (120) The verification results for each M value can be determined based on whether the number of elements is greater than or equal to the threshold value.
[0217] According to the present embodiment, it is possible to verify whether an overlap degree exceeding a certain level is secured, going beyond simply determining whether a single common beam exists between the set of predicted candidate beams and the set of reference beams. Accordingly, it is possible to distinguish between cases where the AI / ML model coincidentally matches a single beam and cases where it reliably predicts multiple valid candidate beams. Furthermore, since the verification strictness can be controlled by adjusting the threshold value, the network can flexibly set AI / ML verification criteria based on service requirements, mobility status, channel variability, or beam management policies.
[0218] In another embodiment, the validation result by M value may include the number of elements in the intersection of the set of prediction candidate beams and the set of metric beams at a specific time instance. In this case, the validation result by M value may be expressed as an integer value or a count value rather than a simple success or failure indicator.
[0219] According to the present embodiment, the degree of overlap between the set of prediction candidate beams and the set of measurement reference beams at a specific time instance can be quantitatively represented. Accordingly, instead of evaluating the beam prediction performance of an AI / ML model merely as success or failure, it is possible to finely determine how closely the Top-K prediction candidate beams match the actual superior Top-M measurement beams. Furthermore, by utilizing the number of elements in the intersection, richer performance information can be provided for comparing prediction performance for different M values, or for AI / ML model selection, model switching, determining alternative actions, or reconstructing validation configuration information.
[0220] In another embodiment, the verification result for each M value may be calculated as an aggregate result for a plurality of time instances. For example, when the number of a plurality of time instances included in the verification interval is N, the second communication node (120) may determine the verification result for each M value based on Equation 2.
[0221]
[0222] In mathematical formula 2, N represents the number of time instances. The meaning of is used with the same meaning as in mathematical formula 1, so the explanation is omitted.
[0223] Referring to mathematical formula 2, ☐ can represent the ratio of time instances where the intersection of the Top-M beam set and the Top-K beam set is not empty (or the ratio of time instances where the verification result can be seen as successful). The second communication node (120) can determine the verification result for each M value based on the number of time instances among multiple time instances where the intersection of the Top-M beam set and the Top-K beam set is not empty. The second communication node (120) can determine the verification result for each M value based on the ratio of time instances among multiple time instances where the intersection of the Top-M beam set and the Top-K beam set is not empty.
[0224] According to the present embodiment, the prediction success frequency of an AI / ML model can be evaluated across multiple time instances without relying on transient success or failure in a single time instance. Accordingly, the instability of verification results caused by instantaneous channel fluctuations, measurement errors, or transient changes in beam quality can be mitigated. Furthermore, by utilizing the number or ratio of time instances with an intersection, AI / ML-based beam management performance can be expressed in the form of a temporally aggregated success rate. This can be utilized as a useful long-term performance indicator for the first communication node to perform model lifecycle management, determine alternative actions, or adjust beam management configurations.
[0225] The verification result for each M value can be determined based on at least one of the number of elements in the intersection between the Top-M beam set and the Top-K beam set among multiple time instances being greater than or equal to a reference value, or the number and ratio of time instances that match the reference value.
[0226] According to the present embodiment, the number of time instances among a plurality of time instances where the degree of overlap between the set of predicted candidate beams and the set of measurement reference beams satisfies a reference level can be calculated. Accordingly, it is possible to distinguish whether the AI / ML model merely includes valid beams occasionally or provides a sufficient number of valid candidate beams reliably at multiple time points. Furthermore, since aggregation at the time instance level using a reference value is possible, the network can apply verification criteria strictly or loosely to match the required beam prediction reliability. This enables AI / ML management operations to be performed by distinguishing between environments requiring high-reliability beam management and environments where candidate beam discovery alone is sufficient.
[0227] The verification result for each M value can be determined based on at least one of the sum of the number of elements in the intersection of the Top-M beam set and the Top-K beam calculated at each of multiple time instances, the average of the number of elements in the intersection, and the ratio of the normalized number of elements in the intersection.
[0228] That is, the verification result for each M value above can be determined based on at least one of the number of elements in the intersection of the Top-M beam set and the Top-K beam set calculated at each of a plurality of time instances, the sum of the number of elements, the average of the number of elements, and the ratio of the normalized number of elements.
[0229] According to the present embodiment, the total overlap between the set of predicted candidate beams and the set of measurement reference beams can be calculated across multiple time instances. Accordingly, not only the success status for each time instance but also the cumulative evaluation of how many actual superior beams the AI / ML model included throughout the entire verification period can be performed. Furthermore, by utilizing the sum of the number of intersection elements, it is possible to distinguish AI / ML models that consistently include more valid candidate beams, even when they have the same number of successes. Therefore, the first communication node can more precisely compare the performance differences between multiple AI / ML models or multiple beam management configurations.
[0230] By performing the above-described step S1340, the first communication node (110) and / or the second communication node (120) can evaluate the relationship between the prediction candidate beam set and the measurement reference beam set in various ways. Accordingly, the first communication node (110) and / or the second communication node (120) can verify AI / ML-based beam management performance based on the degree of overlap between beam sets, the frequency of overlap, and temporal stability, rather than limiting it to a simple threshold comparison method.
[0231] Additionally, since the first communication node (110) and / or the second communication node (120) can produce results of different levels of verification rigor for multiple measurement criteria count M values, the first communication node (110) and / or the second communication node (120) can distinguish and determine whether the set of Top-K prediction candidate beams of the AI / ML model includes the actual top measurement beam, or at least reliably captures the actual excellent measurement beam range. Accordingly, the first communication node (110) can perform verification of the AI / ML model, model lifecycle management, alternative action determination, and reconstruction of beam management configuration information more precisely.
[0232] In step S1350, the second communication node (120) can generate a validation report on AI / ML-based beam management performance based on validation results for each M value. In one embodiment, the validation report may be a composite validation report containing validation results for each M value corresponding to each of the plurality of measurement criteria M. For example, if the M values are {1, 2, 4}, the validation report may include the validation result for M=1, the validation result for M=2, and the validation result for M=4 together.
[0233] A composite verification report can reduce the number of reporting procedures or signaling transactions compared to reporting verification results for each M value individually by including verification results for multiple M values in a single reporting structure. Additionally, since a composite verification report can present verification results from the same or interrelated time instances together, it can be used by the first communication node (110) to determine the performance profile of an AI / ML model by verification rigor level. For example, the first communication node (110) can determine whether the actual top-performing measurement beam is included in the Top-K prediction candidate beam set through the verification results for M=1, and whether the actual excellent candidate measurement beam range overlaps with the Top-K prediction candidate beam set through the verification results for M=2 or M=4.
[0234] In addition to the validation results by M value, the validation report may include at least one of the following: AI / ML model identification information, AI / ML feature identification information, validation configuration identification information, validation time point information, validation interval information, time instance identification information, validation report identification information, information on the number of dimensions M, information on the number of prediction candidates K, prediction candidate beam identifier, dimension beam identifier, intersection beam identifier, number of intersection elements, bitmap, success rate, validation status information, and model management auxiliary information. The specific format of the validation report is described in more detail in FIG. 15, which is described later.
[0235] In one embodiment, the verification report is not limited to a separate dedicated message or dedicated report newly defined for AI / ML verification. For example, the verification report may be implemented as at least one of a performance monitoring report, a verification report, a compressed verification report, or a report composed of a verifiable structure used in the performance monitoring procedure of AI / ML-based beam management.
[0236] Here, a report configured with a verifiable structure may refer to a report containing information configured to determine the relationship between a set of Top-K prediction candidate beams and a set of Top-M metric beams. For example, a report configured with a verifiable structure may include at least one of verification results by M value, success status by M value, number of intersection elements, existence of an intersection, an overlapping bitmap, related beam identifiers, verification time information, verification configuration identification information, or monitoring resource reference information.
[0237] Accordingly, the term "verification report" in this disclosure does not necessarily mean only a novel report format, and existing or separate performance monitoring reports may also be included in the verification report even if they are configured to include information capable of generating or verifying AI / ML verification results.
[0238] In step S1360, if the second communication node (120) requires the transmission of a verification report, it may transmit the verification report to the first communication node (110). In one embodiment, the transmission of the verification report may be triggered by the first communication node (110). For example, the first communication node (110) may provide at least one of a verification report request, report trigger information, report cycle information, and a specific verification event to the second communication node (120).
[0239] In another embodiment, the second communication node (120) may decide for itself whether to transmit the verification report. For example, the second communication node (120) may transmit the verification report when the verification result for each M value satisfies a specific condition, when the verification result is below or above a reference value, when it is determined that the AI / ML-based beam management performance has deteriorated, when the verification period has ended, when the reporting cycle has arrived, or when an internal event of the second communication node (120) occurs.
[0240] The first communication node (110) can perform AI / ML management operations for AI / ML-based beam management based on the received verification report. The AI / ML management operations may include at least one of determining the verification result of an AI / ML model, managing the model lifecycle, determining an alternative action, selecting an AI / ML model, switching AI / ML models, disabling AI / ML functions, reconfiguring AI / ML verification configuration information, reconfiguring beam management configuration information, and requesting retraining of an AI / ML model.
[0241] Figure 14 is a conceptual diagram structurally illustrating the AI / ML verification method described with reference to Figure 13.
[0242] Specifically, FIG. 14 shows a structure for deriving Top-K beam sets and Top-M beam sets from AI / ML inference results and beam monitoring results, respectively, and generating verification results and verification reports based on the relationships between them.
[0243] Referring to FIG. 14, the second communication node (120) can obtain an AI / ML inference result (1401) at least one time instance. The AI / ML inference result (1401) may include at least one of ranking information, priority information, beam identifier information, and beam quality prediction information for beams predicted by an AI / ML model.
[0244] The second communication node (120) can derive a Top-K beam set (1403) based on the AI / ML inference result (1401). Here, the Top-K beam set (1403) can correspond to a prediction candidate beam set.
[0245] Additionally, the second communication node (120) can obtain a beam monitoring result (1402) at least one time instance. The beam monitoring result (1402) may include a beam measurement result for a monitoring resource. For example, the beam monitoring result (1402) may include at least one of L1-RSRP, RSRQ, SINR, and beam quality indicators for CSI-RS, SSB, or other reference signal resources.
[0246] The second communication node (120) can derive a Top-M beam set (1404) based on the beam monitoring result (1402). Here, the Top-M beam set (1404) may correspond to the measurement reference beam set of the claim. The Top-M beam set (1404) may include the top M measurement beams corresponding to the number of measurement references M.
[0247] In one embodiment, when the number of measurement criteria M is set to a single value, one Top-M beam set (1404) can be derived for a specific time instance. In this case, the Top-K beam set (1403) can be compared with the corresponding Top-M beam set (1404).
[0248] In another embodiment, when the number of measurement criteria M is set to a plurality of values, the second communication node (120) can derive a plurality of Top-M beam sets by applying different M values for each M value. In this case, the Top-M beam set corresponding to each M value can be compared with the Top-K beam set.
[0249] The second communication node (120) can generate a verification result (1405) based on the relationship between the Top-K beam set (1403) and the Top-M beam set (1404). In one embodiment, the verification result (1405) may include a verification result for each M value corresponding to each of the plurality of measurement criteria count M. For example, the verification result (1405) may be determined based on at least one of the intersection, inclusion relationship, overlap relationship, and the number of elements of the intersection between the Top-M beam set and the Top-K beam set corresponding to the M value.
[0250] The second communication node (120) can generate a verification report (1406) based on the verification result (1405). In one embodiment, the verification report (1406) may be a composite verification report containing verification results for each M value corresponding to each of the plurality of measurement criteria M. For example, if the M values are set to {1, 2, 4}, the verification report (1406) may include a verification result for M=1, a verification result for M=2, and a verification result for M=4 in a single report structure.
[0251] According to the embodiments of FIGS. 13 and 14, the second communication node (120) can produce verification results for multiple measurement criteria M values, rather than for only one measurement criterion count M. Accordingly, AI / ML-based beam management performance is not evaluated as success or failure based on a single criterion, but can be evaluated as a performance profile based on multiple verification rigor levels.
[0252] Additionally, by using the verification results for multiple M values, the overestimation of performance that may occur when using a single large M value can be reduced. For example, even if the verification result appears as successful for a large M value, it may appear as a failure for a small M value, and in this case, the first communication node (110) may determine that the AI / ML model is not sufficient to capture the actual top measurement beam but is valid in a wider candidate measurement beam range.
[0253] Accordingly, verification based on multiple M values enables a more detailed evaluation of the performance of the AI / ML model and can provide richer judgment information necessary for the first communication node (110) to perform model lifecycle management, alternative action determination, AI / ML model selection, AI / ML model switching, or reconstruction of AI / ML verification configuration information.
[0254] When a value M of multiple measurement criteria is set, the second communication node (120) can produce verification results for different levels of verification rigor by using the same or interrelated AI / ML inference results and beam measurement results. Accordingly, it is possible to evaluate in stages not only whether the set of Top-K predicted candidate beams of the AI / ML model includes the actual top measurement beam, but also the ability to capture the range of measurement beams of actual high quality.
[0255] For example, the verification result for M=1 may indicate whether the Top-K prediction candidate beam set includes the actual top measurement beam, and the verification result for M=2 or M=4 may indicate the prediction candidate coverage performance for the relaxed candidate measurement beam range. Thus, the first communication node (110) can determine the AI / ML-based beam management performance more precisely than when judging the performance of the AI / ML model based on a single verification result.
[0256] In addition, when the verification results for each M value for multiple M values are included in a single composite verification report, the first communication node (110) can obtain performance profiles for multiple verification severity levels through a single reporting procedure. Accordingly, the number of reporting procedures, the number of reconstruction procedures, or the number of signaling transactions can be reduced compared to cases where verification results for each M value are requested or reported individually.
[0257] When only one large metric number M is set, even if the AI / ML model fails to include the actual top metric beam in the set of predicted candidate beams, the validation results may appear good as long as any one within the wide metric beam set overlaps with the set of predicted candidate beams. In this case, it is difficult to distinguish between the AI / ML model's strict top metric beam capture performance and its coverage performance for the candidate beam range, and the model performance may be overestimated.
[0258] On the other hand, when a value M for the number of multiple measurement criteria is set, the second communication node (120) can produce verification results for, for example, M=1, M=2, and M=4, respectively. Accordingly, the first communication node (110) can distinguish whether the AI / ML model is strong at capturing the actual top measurement beam, or whether it is weak at capturing the top measurement beam but suitable for capturing the actual excellent candidate measurement beam range.
[0259] Therefore, verification based on multiple M values has the advantage of being able to profile the beam prediction performance of an AI / ML model at various levels of rigor, compared to a method that verifies using a single large M value with a relaxed standard. As a result, the first communication node (110) can more precisely perform maintenance, switching, retraining requests, alternative action decisions, or reconstruction of beam management configuration information for the AI / ML model.
[0260] FIG. 15 is a block diagram showing the configuration of a verification report. Referring to FIG. 15, the verification report can be implemented as a composite verification report (1500) that includes verification results for each M value for a plurality of measurement criteria M in a single reporting structure.
[0261] The composite verification report (1500) may include at least one of a report header (1501), a verification configuration reference field (1502), a result field by M value (1503), an optional metric field (1504), and a lifecycle instruction field (1505). The names, order, and number of fields shown in FIG. 15 are exemplary, and some fields may be omitted or combined with other fields.
[0262] The report header (1501) may include information identifying the composite verification report (1500) or indicating which AI / ML model, AI / ML function, or verification target the composite verification report (1500) is associated with. For example, the report header (1501) may include at least one of an AI / ML model identifier, an AI / ML function identifier, a beam management function identifier, a verification report identifier, a verification time point identifier, a verification interval identifier, a second communication node identifier, and an association ID. Here, the association ID may be an identifier for associating the composite verification report (1500) with a specific AI / ML verification procedure, a specific verification time point, a specific reporting setting, or specific AI / ML configuration information.
[0263] The verification configuration reference field (1502) may include information indicating which AI / ML verification configuration information the composite verification report (1500) was generated based on. For example, the verification configuration reference field (1502) may include at least one of information on a plurality of measurement criteria count M values, information on a set of M values, information on a prediction candidate count K, information on a K value or a method for determining K, monitoring resource reference information, measurement target reference signal information, verification interval information, and a verification report configuration identifier.
[0264] In one embodiment, the verification configuration reference field (1502) may indicate that the set of M values is set to {1, 2, 4}. In another embodiment, the verification configuration reference field (1502) may indicate whether the number of prediction candidates K is applied commonly to multiple M values, or is set per M value depending on the M value. In yet another embodiment, the verification configuration reference field (1502) may indicate which set of CSI-RS, SSB, or monitoring resources the composite verification report (1500) was generated based on measurement results.
[0265] The result field (1503) for each M value may include a verification result for each M value corresponding to each of the multiple measurement criteria count M. For example, the result field (1503) for each M value may include a verification result for M=1, a verification result for M=2, and a verification result for M=4. Each verification result for each M value may be calculated based on the relationship between the set of measurement criteria beams and the set of prediction candidate beams corresponding to the M value.
[0266] In one embodiment, the result field (1503) for each M value may include an indicator indicating success or failure for each M value. For example, the result field (1503) for each M value may indicate whether the verification result for M=1 is successful or failed, whether the verification result for M=2 is successful or failed, and whether the verification result for M=4 is successful or failed, respectively. In another embodiment, the result field (1503) for each M value may represent the verification result for each M value as at least one of a bitmap, an index, a code point, a count value, and a ratio value.
[0267] The optional metric field (1504) may include additional information that supports the validation results included in the result field (1503) by M value or indicates details of the validation results. For example, the optional metric field (1504) may include at least one of the number of intersection elements between the prediction candidate beam set and the metric beam set, the intersection ratio, the overlap ratio, the intersection beam identifier, the overlap bitmap, the number of successes, the success ratio, the cumulative number of intersection elements for multiple time instances, and the number of times the threshold value is satisfied.
[0268] The optional metric field (1504) does not always need to be included. For example, to reduce reporting overhead, the result field (1503) by M value may only include success or failure, and the optional metric field (1504) may be omitted. Conversely, if the first communication node (110) requires more detailed performance analysis, the optional metric field (1504) may include additional metrics such as the number of intersection elements, the intersection ratio, and a bitmap.
[0269] The lifecycle instruction field (1505) may include information that can be used by the first communication node (110) to perform AI / ML management operations based on the composite verification report (1500). For example, the lifecycle instruction field (1505) may include at least one of the following: verification status, verification success status, verification failure status, need to maintain the model, need to switch the model, need to retrain the model, need to disable AI / ML functions, alternative operation assistance information, assistance information for reconfiguring AI / ML verification configuration information, and assistance information for reconfiguring beam management configuration information.
[0270] In one embodiment, the lifecycle indicator field (1505) may include a verification status determined by the second communication node (120). In another embodiment, the lifecycle indicator field (1505) may include auxiliary information necessary for the first communication node (110) to determine an alternative action, such as a transition from AI / ML-based beam management to conventional beam measurement-based beam management. In yet another embodiment, the lifecycle indicator field (1505) may include status information indicating that, considering the verification results for a plurality of M values, the AI / ML model is insufficient for capturing the actual top-level measurement beam but is valid for a wider range of candidate measurement beams.
[0271] The composite verification report (1500) can provide verification results for multiple M values in a single report structure using the aforementioned fields. Accordingly, the first communication node (110) can check the performance profile of the AI / ML model by verification rigor level through the composite verification report (1500), and can perform verification of the AI / ML model, model lifecycle management, alternative action determination, AI / ML model selection, AI / ML model switching, and reconstruction of beam management configuration information more precisely.
[0272] FIG. 16 illustrates an AI / ML verification method performed by a first communication node (110). The procedure of FIG. 16 may be a network-side (first communication node (110) side) procedure corresponding to the verification report generation and transmission procedure of the second communication node (120) described in FIG. 13 to 15.
[0273] Referring to FIG. 16, in step S1610, the first communication node (110) can transmit AI / ML verification configuration information to the second communication node (120). The AI / ML verification configuration information may be information for verifying AI / ML-based beam management performance. For example, the AI / ML verification configuration information may include at least one of information on the number of multiple measurement criteria M values, information on the number of prediction candidates K, monitoring resource information, information on the AI / ML model or AI / ML function to be verified, information on the verification time or verification interval, and information on the reporting conditions and reporting format of the verification report.
[0274] In one embodiment, the number of prediction candidates K can be set to a fixed value regardless of the number of multiple measurement criteria M. In another embodiment, the number of prediction candidates K can be set for each value of M depending on the number of measurement criteria M.
[0275] In step S1620, the first communication node (110) may receive a verification report from the second communication node (120). The verification report may include verification results for AI / ML-based beam management performance. In one embodiment, the verification report may be a composite verification report containing verification results for each M value corresponding to each of the plurality of measurement criteria M.
[0276] For example, a composite verification report may include verification results for M=1, verification results for M=2, and verification results for M=4 in a single report structure. The verification result for each M value may be calculated based on the relationship between the Top-K prediction beam set derived based on AI / ML inference results and the Top-M measurement beam set of the corresponding M value derived based on beam monitoring results. Accordingly, the first communication node (110) can verify the AI / ML-based beam management performance according to different measurement criteria number M together.
[0277] In step S1630, the first communication node (110) can utilize the received verification report. For example, based on the verification report, the first communication node (110) can perform at least one of determining the verification result of the AI / ML model, managing the model lifecycle, determining alternative actions, selecting the AI / ML model, switching the AI / ML model, disabling the AI / ML function, reconfiguring the AI / ML verification configuration information, and reconfiguring the beam management configuration information.
[0278] In one embodiment, the first communication node (110) can determine the performance profile of the AI / ML model using verification results for a plurality of M values. For example, the verification result for a small M value may be a relatively strict verification result indicating whether the AI / ML model has included the top beam or very good beam in the predicted Top-K beam set in actual measurements. On the other hand, the verification result for a large M value may be a relatively relaxed verification result indicating whether the AI / ML model has included at least some of the top candidate beams in actual measurements in the predicted Top-K beam set.
[0279] Accordingly, the first communication node (110) can determine, based on the verification report, whether the AI / ML model is suitable for accurately capturing the top measurement beam or for providing a valid prediction beam within a wide set of measurement candidate beams. Accordingly, the first communication node (110) can decide whether to maintain the AI / ML model, switch to another model, perform an alternative action, or reset the verification configuration information.
[0280] According to the embodiment of FIG. 16, the first communication node (110) can obtain verification results for multiple measurement criteria M values through a single verification report, so that AI / ML-based beam management performance can be evaluated at the level of multiple measurement criteria widths or verification granularity rather than a single criterion. Accordingly, the overestimation of AI / ML model performance that may occur when using only one large M value can be reduced, and model lifecycle management and beam management configuration control can be performed more precisely.
[0281] The following describes an example of verifying AI / ML model updates or local training results in an FL / DL environment. Here, FL may mean federated learning, and DL may mean distributed learning. In an FL / DL environment, a second communication node (120) can perform local training, fine-tuning, or model updates of an AI / ML model based on local data or field measurement data, and a first communication node (110) can perform global model updates, model aggregation, or AI / ML model lifecycle management based on local update information received from a plurality of second communication nodes.
[0282] However, if local update information generated at the second communication node (120) is uploaded to the first communication node (110) or reflected in the global model aggregation while lacking sufficient performance, the performance of the global model may be degraded. Therefore, the present embodiment proposes a structure in which the second communication node (120) performs local verification after local training or model update, and transmits a report message or model update-related information only when the result of the local verification satisfies a predetermined condition.
[0283] FIG. 17 illustrates a local verification and model update reporting procedure performed by a second communication node (120) in an FL / DL environment.
[0284] Referring to FIG. 17, in step S1701, the second communication node (120) can obtain FL / DL verification configuration information from the first communication node (110). The FL / DL verification configuration information may be configuration information for controlling FL / DL-based AI / ML model updates, local training, or local model verification.
[0285] FL / DL validation configuration information may include at least one of information regarding local validation conditions, local validation KPIs, local validation thresholds, local training conditions, model update reporting conditions, model delta reporting conditions, compression KPI reporting conditions, model status reporting conditions, or reporting message formats. For example, FL / DL validation configuration information may instruct to report model update information only when the local validation KPI is greater than or equal to the threshold.
[0286] Here, the local validation KPI may be an indicator representing the quality of local training or model updates performed at the second communication node (120). For example, the local validation KPI may include at least one of accuracy, loss value, inference confidence, beam prediction performance, Top-M / Top-K based validation results, latency, model stability indicator, or use-case specific KPI.
[0287] In one embodiment, FL / DL verification configuration information may be provided to a second communication node (120) through at least one of the RRC layer, MAC layer, or PHY layer. For example, local verification conditions, local verification threshold values, model delta reporting conditions, or KPI reporting conditions may be provided as RRC configuration information, and specific reporting triggers, reporting timing, or uplink resources may be linked to MAC or PHY layer procedures. Accordingly, the present embodiment may provide a structure that manages the verification and reporting of local updates in a wireless access network environment in conjunction with communication procedures, rather than a simple FL / DL algorithm itself.
[0288] In step S1702, the second communication node (120) may perform local training. Local training may be performed based on at least one of local data, measurement data, channel state information, beam measurement results, user mobility information, or service environment information held or acquired by the second communication node (120). Local training may include actions to update some parameters of an AI / ML model, actions to fine-tune the model, actions to generate a model delta, or actions to update the local model state.
[0289] In step S1703, the second communication node (120) can evaluate local verification KPIs. The second communication node (120) can perform local verification on the local model, model update information, or model delta generated by local training, and produce local verification KPIs as a result.
[0290] For example, the second communication node (120) can evaluate the beam prediction performance of the updated AI / ML model using local validation data or recent measurement data. In another example, the second communication node (120) can calculate local validation KPIs based on the relationship between the set of Top-K prediction candidate beams and the set of Top-M measurement reference beams described in FIGS. 13 through 16.
[0291] After step S1703, the second communication node (120) can determine whether the local verification KPI is greater than or equal to the local threshold value. In FIG. 17, this is illustrated as KPI_local >= theta_local. Here, KPI_local may refer to the local verification KPI calculated by the second communication node (120), and theta_local may refer to the local verification threshold value set by FL / DL verification configuration information or predefined.
[0292] If the local verification KPI is greater than or equal to the local threshold value, the second communication node (120) can generate and transmit a report message in step S1704. The report message may be a message to transmit local training or model update results to the first communication node (110).
[0293] In one embodiment, the report message may include at least one of modelDeltaUpdate, compressedKPIReport, modelStatusReport, or FederatedLearningStatusReport.
[0294] modelDeltaUpdate may be an information element or message regarding model delta information indicating model parameters, weight differences, gradient information, or the amount of model update changed by local training. For example, instead of transmitting all AI / ML model parameters, modelDeltaUpdate may represent the difference value with the previous model state or the global model state. Accordingly, the second communication node (120) can provide the information necessary for model updating to the first communication node (110) more efficiently.
[0295] A compressedKPIReport can be an information element or message intended to report local validation KPIs or model performance metrics in a compressed form. For example, a compressedKPIReport does not include the entire validation log or full measurement results, but may include representative KPIs, averages, cumulative values, ratios, quantized KPI values, and KPI information in the form of bitmaps or code points. This allows for reporting local validation results while reducing signaling overhead.
[0296] modelStatusReport may be an information element or message for reporting the local model status of the second communication node (120). For example, modelStatusReport may include at least one of a local model identifier, a global model version, a local training identifier, a model update time, a model validity status, a model validation status, whether the model is applicable, or a model synchronization status.
[0297] FederatedLearningStatusReport may be an information element or message for reporting the status of an FL / DL procedure. For example, FederatedLearningStatusReport may include at least one of whether local training is complete, whether local validation is successful, whether model delta upload is possible, whether aggregation participation is possible, FL session status, DL session status, local data availability, reporting delay status, or whether additional configuration information is requested.
[0298] In addition to the information elements described above, the report message may further include at least one of a local validation KPI, a local threshold value, a validation result, a model identifier, a function identifier, a report identifier, a local training identifier, global model version information, model delta size information, compression method information, or status information of a second communication node.
[0299] In one embodiment, the second communication node (120) may transmit `modelDeltaUpdate` based on local verification results instead of transmitting the entire model parameters. Additionally, the second communication node (120) may transmit local verification KPIs in a compressed form via `compressedKPIReport` instead of transmitting the entire KPI log or the entire verification log. For example, `compressedKPIReport` may include average KPIs, representative KPIs, quantized KPI values, code points, bitmaps, or verification status indicators.
[0300] In one embodiment, the first communication node (110) or the second communication node (120) may adjust the reporting period or reporting frequency based on the `adaptiveReportingRate`. For example, the reporting frequency may be reduced when the local verification results are stable, and the reporting frequency may be increased when the local verification results are close to a threshold value or fluctuate rapidly. Accordingly, unnecessary reporting can be reduced and RAN signaling overhead can be mitigated even when there are many UEs or model updates occur frequently in an FL / DL environment.
[0301] In step S1707, the first communication node (110) may perform network-side updates based on a report message. Network-side updates may be performed based on model deltas, compressed KPIs, model status information, or FL / DL status information received from the second communication node (120). For example, the first communication node (110) may determine whether to reflect the received modelDeltaUpdate in the global model aggregation, or whether to include the local update information of the corresponding second communication node (120) in the aggregation based on the compressedKPIReport. Additionally, the first communication node (110) may update the global model status, model lifecycle status, or FL / DL session status based on the modelStatusReport or FederatedLearningStatusReport.
[0302] The first communication node (110) can determine whether to include the local update in the global model aggregation based on at least one of the received local validation result, `compressedKPIReport`, `modelStatusReport`, or `modelDeltaUpdate`. For example, the first communication node (110) may allow the aggregation of the local update if the local validation KPI satisfies a threshold value, and may block the aggregation of the local update or require additional validation if the local validation KPI does not satisfy a threshold value.
[0303] Additionally, the first communication node (110) can determine at least one of whether to allow model delta upload, whether to reflect global model updates, whether to continue local training, whether to stop local training, whether to change the model lifecycle state, whether to roll back the model, whether to decide on an alternative action, whether to perform edge caching, or whether to reset the validation threshold value based on the report message.
[0304] On the other hand, if the local verification KPI does not satisfy the local threshold value, the second communication node (120) may omit network-side updates in step S1708. Here, omitting network-side updates may include the second communication node (120) not transmitting model update information, model delta, or local model parameters generated by local training to the first communication node (110), or ensuring that such information is not reflected in the global model aggregation.
[0305] That is, in step S1708, it is possible to prevent low-quality model update information that does not satisfy local verification criteria from being uploaded to the first communication node (110) or reflected in the global model. Accordingly, the contamination of the global model by low-quality local updates in the FL / DL environment can be reduced.
[0306] In step S1709, the second communication node (120) may continue local training or request new configuration information from the first communication node (110). For example, if the local validation KPI does not satisfy the local threshold value, the second communication node (120) may continue local training based on additional local data or request different training parameters, different validation threshold values, different reporting conditions, or different AI / ML model configurations.
[0307] In one embodiment, the second communication node (120) may transmit a new configuration information request message to the first communication node (110). The new configuration information request message may include at least one of the cause of local verification failure, local verification KPI, current model status, type of requested configuration information, whether additional learning is required, or whether to retry the report.
[0308] In one embodiment, the second communication node (120) may decide whether to continue local training, delay reporting, or request new FL / DL verification configuration information based on the mobilityState, channel state, or verification result. For example, if the mobility of the second communication node (120) is high or the channel state fluctuates rapidly, the second communication node (120) may determine that the reliability of the local verification result is low and request additional local verification or new configuration information.
[0309] According to the embodiment of FIG. 17, the second communication node (120) does not unconditionally report local training results to the first communication node (110) or reflect them in the global model aggregation, but can control whether to report or reflect updates based on local verification KPIs. Accordingly, the reflection of low-quality local updates in the global model can be reduced, and the reliability of FL / DL-based AI / ML model management can be improved.
[0310] Additionally, according to the embodiment of FIG. 17, the second communication node (120) may selectively report necessary model update information and validation status information using information elements or messages such as modelDeltaUpdate, compressedKPIReport, modelStatusReport, or FederatedLearningStatusReport, without reporting all model parameters or all validation logs. Accordingly, signaling overhead related to model updates in an FL / DL environment can be reduced.
[0311] FIG. 18 illustrates a structure that synchronizes local model status and global model status in an FL / DL environment and performs FL / DL-based AI / ML model management operations based on status and KPI reporting.
[0312] What is important in the embodiment of FIG. 18 is not merely reporting or storing model identifiers, but managing the correspondence and synchronization status between the local model state of the second communication node (120) and the global model state managed by the first communication node (110). That is, the first communication node (110) can determine, using localTrainingID, globalModelVersion, and model ID mapping table (1808), which global model version a specific local update was generated based on, which local training procedure the local update corresponds to, and whether the local update can be reflected in the global model aggregation.
[0313] Referring to FIG. 18, the first communication node (110) can manage a global model registry (1802). The global model registry (1802) may be a configuration that stores or manages identification information, version information, applicable AI / ML function information, model status information, or model lifecycle information for a global AI / ML model. For example, the global model registry (1802) may include at least one of a globalModelVersion, a global model identifier, an AI / ML function identifier, a model deployment status, a model update status, or a model validity status.
[0314] The adaptive control module (1803) can perform FL / DL-related control for the second communication node (120) based on at least one of the global model registry (1802), status and KPI reporting (1807), and model ID mapping table (1808). For example, the adaptive control module (1803) can determine at least one of local validation conditions, local validation threshold values, local training conditions, model delta reporting conditions, KPI reporting cycle, compressed KPI reporting method, whether to allow model updates, or whether to provide new configuration information.
[0315] The second communication node (120) can manage a local training context (1805) and a local model state (1806). The local training context (1805) may include information related to local training, fine-tuning, or model updates performed at the second communication node (120). For example, the local training context (1805) may include at least one of a localTrainingID, local data state, number of training sessions, training intervals, type of local data used, training parameters, local validation conditions, or local training progress status.
[0316] The local model state (1806) may represent the state of a local AI / ML model held or executed by the second communication node (120). For example, the local model state (1806) may include at least one of a local model identifier, a local model version, a local model parameter state, whether the local model has been updated, whether local verification was successful, whether the local model is valid, or a synchronization state with a global model.
[0317] Status and KPI reporting (1807) may refer to an action or report message in which the second communication node (120) reports information regarding the local training context (1805), local model status (1806), local validation result, or local validation KPI to the first communication node (110). In one embodiment, status and KPI reporting (1807) may include at least one of modelStatusReport, compressedKPIReport, FederatedLearningStatusReport, or modelDeltaUpdate.
[0318] modelStatusReport may be an information element or message for reporting local model status (1806). For example, modelStatusReport may include at least one of a local model identifier, a global model version, a local model version, a local validation status, model applicability, or model synchronization status.
[0319] A compressedKPIReport may be an information element or message for reporting local validation KPIs or model performance metrics in a compressed form. For example, a compressedKPIReport may include accuracy, loss values, success rates, Top-M / Top-K based validation results, quantized KPI values, and KPI information in the form of bitmaps or code points. Accordingly, the second communication node (120) can efficiently provide local validation results to the first communication node (110) without reporting the entire validation log or entire measurement results.
[0320] FederatedLearningStatusReport may be an information element or message for reporting the status of an FL / DL procedure. For example, FederatedLearningStatusReport may include at least one of whether local training is complete, whether participation in aggregation is possible, whether model delta upload is possible, FL / DL session status, local data availability, reporting delay status, or whether additional configuration information is requested.
[0321] modelDeltaUpdate can be an information element or message representing model parameters, weight differences, gradient information, or the amount of model updates changed by local training. For example, instead of transmitting all model parameters, modelDeltaUpdate can be used to transmit difference information with respect to a global model or a previous local model. Accordingly, RAN signaling overhead related to model updates can be reduced in an FL / DL environment.
[0322] The model ID mapping table (1808) may be a table that manages identifier or version relationships between a global model and a local model. For example, the model ID mapping table (1808) may include at least one of the correspondence between a global model identifier and a local model identifier, the correspondence between globalModelVersion and a local model version, the correspondence between localTrainingID and a model delta, and the correspondence between an AI / ML feature identifier and an applied model.
[0323] In one embodiment, the second communication node (120) may report at least one of localTrainingID and globalModelVersion to the first communication node (110) via modelStatusReport. Based on the received localTrainingID and globalModelVersion, the first communication node (110) may update or verify the mapping between the local model identifier and the global model identifier. Accordingly, the inclusion of model deltas generated based on an incorrect global model version in the aggregation may be reduced, and discrepancies between the local model status and the global model status may be mitigated.
[0324] In one embodiment, the first communication node (110) can update the model ID mapping table (1808) after receiving status and KPI reporting (1807). For example, the first communication node (110) can check through the model ID mapping table (1808) which global model version the local model of the second communication node (120) was trained on, which localTrainingID the local model update corresponds to, and whether the model delta can be reflected in the global model aggregation.
[0325] Additionally, the adaptive control module (1803) can generate control information for the second communication node (120) based on the model ID mapping table (1808) and status / KPI reporting (1807). For example, the adaptive control module (1803) can control the reflection of the local update in the global model update if the local verification KPI is sufficient, and can control the provision of new FL / DL verification configuration information, changed local verification threshold values, or other model configuration information to the second communication node (120) if the local verification KPI is insufficient.
[0326] In one embodiment, the adaptive control module (1803) can adjust the reporting cycle or reporting frequency of the second communication node (120) based on the adaptiveReportingRate. For example, the reporting frequency may be reduced when the local verification result is stable, and the reporting frequency may be increased when the local verification result approaches a threshold value or fluctuates rapidly. Accordingly, even when the number of second communication nodes is large or model updates occur frequently in an FL / DL environment, unnecessary reporting can be reduced and RAN signaling overhead can be mitigated.
[0327] In one embodiment, the adaptive control module (1803) may further consider at least one of the mobilityState, channel state, handover state, whether the serving cell has changed, or whether the serving edge node has changed. For example, if the mobility of the second communication node (120) is high or a handover is expected, the adaptive control module (1803) may adjust the local verification threshold value, increase the reporting frequency, or trigger a procedure for transmitting or maintaining the local model state and model delta.
[0328] In one embodiment, when a handover or serving node change occurs, the first communication node (110) can control that at least one of the local training state, model delta, validation result, or model state information is transmitted to a new serving cell or a new edge node. Accordingly, the continuity of the FL / DL session, i.e., flDlSessionContinuity, can be maintained even if the second communication node (120) moves.
[0329] In one embodiment, the first communication node (110) or edge node may store local model, model delta, validation results, compressed KPI, or model state information based on edge caching. For example, when the second communication node (120) moves from a specific edge node to another edge node, the local training state or model validation results stored in the previous edge node may be reused at the new edge node. Accordingly, local training may be re-performed from the beginning or the same validation may be repeated even after a handover or edge node change.
[0330] The first communication node (110) can determine whether to include the local update in the global model aggregation based on the status and KPI reporting (1807). For example, the first communication node (110) may allow the aggregation of the local update if the local verification KPI satisfies a threshold value, and may block the aggregation of the local update or require additional verification if the local verification KPI does not satisfy a threshold value.
[0331] Additionally, the first communication node (110) can determine at least one of whether to allow model delta uploads, whether to reflect global model updates, whether to adjust the reporting cycle per UE, whether to continue local training, whether to stop local training, whether to change the model lifecycle state, whether to roll back the model, whether to decide on an alternative action, whether to perform edge caching, or whether to reset the validation threshold value, based on the status and KPI reporting (1807).
[0332] According to the embodiment of FIG. 18, the first communication node (110) can reduce the discrepancy between the global model state and the local model state by using the local training context (1805), local model state (1806), and state / KPI reporting (1807) of the second communication node (120). Additionally, by using the model ID mapping table (1808), it is possible to clearly manage which global model version the local model update is based on, and reduce the reflection of incorrect model deltas in the global model aggregation.
[0333] Furthermore, according to the embodiment of FIG. 18, the local verification results may not be limited to simple reporting information but may be connected to actual FL / DL-based AI / ML model management operations. For example, the first communication node (110) may determine, based on the status and KPI reporting (1807), whether to include global aggregation of local updates, whether to require additional verification, whether to change the model lifecycle status, whether to allow model delta uploads, whether to adjust the reporting cycle, whether to continue or stop local training, whether to roll back the model, whether to perform an alternative action, whether to perform edge caching, or whether to reset the verification threshold value.
[0334] Accordingly, it is possible to reduce the contamination of global models by low-quality local updates in FL / DL environments, improve FL / DL session continuity even in mobility or handover situations, and reduce RAN signaling overhead associated with model update and validation reporting.
[0335] An AI / ML verification method according to exemplary embodiments has been described above with reference to FIGS. 1 to 18.
[0336] According to at least one embodiment, AI / ML-based beam management performance can be verified based on the relationship between the Top-K predicted candidate beam set derived from AI / ML inference results and the Top-M measurement reference beam set derived from beam measurement results. Accordingly, it is possible to directly evaluate whether the AI / ML model includes high-quality measurement beams in the predicted candidate beam set in an actual wireless environment.
[0337] According to at least one embodiment, a verification result can be produced based on the intersection, inclusion relationship, overlap relationship, or the number of elements in the intersection between the Top-K prediction candidate beam set and the Top-M metric beam set. Accordingly, AI / ML-based beam prediction performance, which is difficult to express solely through a comparison method of simple scalar values and thresholds, can be effectively verified based on the relationships between beam sets.
[0338] According to at least one embodiment, verification results for each M value can be produced for a plurality of measurement criteria M values. Accordingly, the performance of an AI / ML model can be assessed stepwise at different measurement criterion widths or verification granularity levels.
[0339] According to at least one embodiment, unlike the case where only one large M value is used, the verification results for a small M value and the verification results for a large M value can be checked together. Accordingly, it is possible to distinguish whether the AI / ML model includes the actual top measurement beam within the prediction candidate or includes a valid beam among a wider set of measurement candidate beams within the prediction candidate, and the overestimation of the performance of the AI / ML model can be reduced.
[0340] According to at least one embodiment, at least one of the AI / ML inference results and beam measurement results obtained in the same or interrelated time instances can be reused for verification of multiple M values. Accordingly, even when verification of multiple M values is performed, additional AI / ML inference or additional beam measurements can be reduced, and the computational complexity and measurement burden of the second communication node can be alleviated.
[0341] According to at least one embodiment, verification results for a plurality of M values can be included in a single composite verification report. Accordingly, the number of reporting procedures, the number of reconstruction procedures, or the number of signaling transactions can be reduced compared to cases where verification results for each M value are requested or reported individually.
[0342] Based on the foregoing description of various embodiments of this disclosure, a person skilled in the art will clearly understand that the methods and / or processes of the present invention and the steps thereof may be realized in hardware, software, or any combination of hardware and software suitable for a particular use case. The hardware may include a general-purpose computer and / or a dedicated computing device or a specific computing device or a particular form or component of a specific computing device. The processes may be realized by one or more processors having internal and / or external memory, such as a microprocessor, a controller, such as a microcontroller, an embedded microcontroller, a microcomputer, an arithmetic logic unit (ALU), a digital signal processor, such as a programmable digital signal processor, or other programmable device. In addition, or as an alternative, the above processes may be carried out by an application-specific integrated circuit (ASIC), a programmable gate array, such as a field programmable gate array (FPGA), a programmable logic unit (PLU), or a programmable array logic (PAL), or any other device capable of executing and responding to instructions, any other device or combination of devices that may be configured to process electronic signals. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software.For ease of understanding, the processing unit may be described as being used as a single unit, but a person of ordinary skill in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.
[0343] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more machine-readable recording media.
[0344] Furthermore, parts contributing to the objects of the technical solution of the present invention or to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-readable medium. The machine-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the machine-readable recording medium may be those specifically designed and configured for the embodiments, or they may be those known and available to a person skilled in the art of computer software. Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs, DVDs, and Blu-rays; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, bytecode, as well as high-level language code that can be executed by a computer using an interpreter, etc., which can be created using a structured programming language such as C, an object-oriented programming language such as C++, or high-level or low-level programming languages (assembly language, hardware description languages, and database programming languages and technologies), which can be stored and compiled or interpreted to be executed on a machine capable of executing any of the aforementioned devices, as well as a processor, a processor architecture, or a heterogeneous combination of different hardware and software combinations.
[0345] Accordingly, in one embodiment according to the present invention, when the methods and combinations thereof described above are performed by one or more computing devices, the methods and combinations thereof may be implemented as executable code that performs each step. In another embodiment, the methods may be implemented as systems that perform the steps, and the methods may be distributed in various ways across devices, or all functions may be integrated into a single dedicated, standalone device or other hardware. In yet another embodiment, the means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.
[0346] For example, the above-described hardware device may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, or TPU that is combined with memory such as ROM / RAM for storing program instructions and configured to execute instructions stored in said memory, and may include a communication unit capable of exchanging signals with an external device. Additionally, the hardware device may include a keyboard, mouse, or other external input device for receiving instructions written by developers.
[0347] Although the present invention has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments, and a person skilled in the art to which the invention belongs can make various modifications and variations from this description.
[0348] Accordingly, the scope of the present invention is not limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims attached to this disclosure, as well as the claims attached to this disclosure, shall be considered to be within the scope of the scope of the concept of the present invention. For example, appropriate results may be achieved even if the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from the described method, or are replaced or substituted by other components or equivalents.
[0349] Such equivalent or equivalent modifications may include, for example, logically equivalent methods capable of producing the same result as carrying out the method according to the present invention; the true meaning and scope of the present invention shall not be limited by the examples described above, but shall be understood in the broadest sense permissible by law.
Claims
1. A method for verifying AI / ML in a wireless communication system performed by a second communication node including a communication unit and a processor, A step of obtaining AI / ML verification configuration information from a first communication node; A step of deriving a set of measurement reference beams corresponding to each of a plurality of measurement reference counts M based on beam results measured in at least one time instance; A step of deriving a set of prediction candidate beams corresponding to the number of prediction candidates K based on AI / ML inference results in at least one time instance; For each of the plurality of measurement criteria M, a step of determining a verification result for each M value by evaluating the relationship between the measurement criterion beam set corresponding to the measurement criterion M and the prediction candidate beam set; It includes the step of generating a verification report on AI / ML-based beam management performance based on the verification results for each M value above, The above AI / ML verification configuration information is an AI / ML verification method that includes information on the number of M values of the plurality of measurement criteria.
2. In Paragraph 1, The above AI / ML validation configuration information further includes information regarding the number of prediction candidates K, and An AI / ML validation method in which the number of predicted candidates K is applied as a fixed value regardless of the number of multiple measurement criteria M.
3. In Paragraph 1, The above prediction candidate count K is an AI / ML verification method in which, for each of the above plurality of measurement criteria count M, it is set according to the value of M depending on the corresponding measurement criteria count M.
4. In Paragraph 1, An AI / ML verification method in which at least one of the AI / ML inference result and the beam measurement result obtained at each of the above at least one time instance is reused to derive at least one of the measurement beam set for each of the above prediction candidate beam set and the above plurality of measurement criteria number M.
5. In Paragraph 1, AI / ML verification method, wherein the above verification report is generated as a composite verification report including verification results for each M value corresponding to each of the plurality of measurement criteria M.
6. In Paragraph 1, An AI / ML verification method in which the verification result for each M value is determined based on at least one of the intersection, inclusion relationship, overlap relationship between the measurement reference beam set corresponding to the number of measurement references M and the prediction candidate beam set, and the number of elements of the intersection.
7. In Paragraph 6, An AI / ML validation method in which the validation result for each M value is determined based on whether the intersection of the set of measurement beams corresponding to the number of measurement criteria M in a specific time instance and the set of prediction candidate beams is an empty set.
8. In Paragraph 6, An AI / ML verification method in which the verification result for each M value is determined based on whether the number of elements in the intersection of the metric beam set corresponding to the number of metric M in a specific time instance and the prediction candidate beam set is greater than or equal to a threshold value.
9. In Paragraph 6, An AI / ML verification method in which the above verification result for each M value includes the number of elements of the intersection of the metric beam set corresponding to the number of metric M in a specific time instance and the prediction candidate beam set.
10. In Paragraph 6, An AI / ML verification method in which the verification result for each M value is determined based on at least one of the number and ratio of time instances in which the intersection of the measurement reference beam set corresponding to the number of measurement references M among a plurality of time instances and the prediction candidate beam set is not an empty set.
11. In Paragraph 6, An AI / ML verification method in which the verification result for each M value is determined based on the number of time instances in which the number of elements in the intersection of the measurement reference beam set corresponding to the number of measurement references M among a plurality of time instances and the prediction candidate beam set is greater than or equal to the reference value or matches the reference value.
12. In Paragraph 6, AI / ML verification method, wherein the verification result for each M value is determined based on at least one of the sum, average, and normalized ratio of the number of elements of the intersection of the measurement beam set corresponding to the number of measurement criteria M determined in each of a plurality of time instances and the prediction candidate beam set.
13. In a second communication node comprising a communication unit and a processor, The processor performs the steps of: obtaining AI / ML verification configuration information from a first communication node; deriving a set of measurement reference beams corresponding to each of a plurality of measurement reference counts M based on beam results measured at at least one time instance; deriving a set of prediction candidate beams corresponding to a number of prediction candidates K based on AI / ML inference results at at least one time instance; for each of the plurality of measurement reference counts M, evaluating the relationship between the set of measurement reference beams corresponding to the measurement reference count M and the set of prediction candidate beams to determine a verification result for each M value; and generating a verification report on AI / ML-based beam management performance based on the verification result for each M value, wherein the AI / ML verification configuration information includes information on the plurality of measurement reference count M values, and the second communication node.
14. A method for verifying AI / ML in a wireless communication system performed by a first communication node including a communication unit and a processor, A step of providing AI / ML verification configuration information to a second communication node; A step of receiving a verification report on AI / ML-based beam management performance from the second communication node; and It includes a step of performing AI / ML operation control based on the above verification report, The above AI / ML validation configuration information includes information on the number of multiple measurement criteria M values, and The above verification report is generated based on the verification results for each M value corresponding to each of the plurality of measurement criteria M, and An AI / ML verification method in which the verification result for each M value is determined by evaluating the relationship between a set of prediction candidate beams corresponding to the number of prediction candidates K derived based on AI / ML inference results in at least one time instance and a set of measurement reference beams corresponding to the number of measurement references M derived based on beam results measured in at least one time instance.
15. In Paragraph 14, The above AI / ML operation control is an AI / ML verification method comprising at least one of determining the verification result of an AI / ML model, managing the lifecycle of an AI / ML model, returning to a non-AI / ML state, disabling AI / ML functions, and reconfiguring AI / ML verification configuration information.
16. In Paragraph 14, The above AI / ML validation configuration information further includes information regarding the number of prediction candidates K, and An AI / ML validation method in which the number of predicted candidates K is applied as a fixed value regardless of the number of multiple measurement criteria M.
17. In Paragraph 14, AI / ML verification method, wherein the above verification report is a composite verification report comprising verification results for each M value corresponding to each of the plurality of measurement criteria M.
18. In Paragraph 14, An AI / ML verification method in which the verification result for each M value is determined based on at least one of the intersection, inclusion relationship, overlap relationship between the measurement reference beam set corresponding to the number of measurement references M and the prediction candidate beam set, and the number of elements of the intersection.
