Methods and systems for estimating response times for processing of service requests
The Response Time Estimator Function addresses the variability of LLM-NF response times by calculating MICT, improving network resource management and user experience through accurate timing estimates.
Patent Information
- Application Number
- PCT/EP2025/063378
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-19
AI Technical Summary
The variability and non-determinism of response times in Large Language Model (LLM)-based Network Functions (LLM-NFs) pose challenges for systems relying on timely responses, leading to service failures, degraded user experience, or inefficient resource utilization due to premature timeouts or excessive delays.
A Response Time Estimator Function (RTEF) is introduced to calculate an estimated Model Inference Completion Time (MICT) for LLM-NFs, considering model and hardware parameters, input/output sequence limits, and historical data to provide accurate timing estimates, enabling informed decision-making by consumer entities.
Enhances network resource management, prevents unnecessary waiting or timeouts, and fosters more predictable and reliable interactions with LLM-NFs by providing precise response time forecasts.
Smart Images

Figure EP2025063378_19112026_PF_FP_ABST
Abstract
Description
[0001] METHODS AND SYSTEMS FOR ESTIMATING RESPONSE TIMES FOR PROCESSING OF SERVICE REQUESTS
[0002] TECHNICAL FIELD
[0003] The present disclosure relates, in general, to network function management and, more specifically, although not exclusively to methods and systems for estimating and managing the response times of network functions that utilize Large Language Models (LLMs).
[0004] BACKGROUND
[0005] Large Language Models have demonstrated remarkable capabilities in various natural language processing tasks and are increasingly being integrated into diverse applications, including as components of network functions within telecommunications and other networked systems. These LLM-based Network Functions (LLM-NFs) can perform complex tasks such as intelligent routing, content generation, anomaly detection, and sophisticated user interaction. However, a significant operational challenge associated with LLM-NFs is the inherent variability and non-determinism of their response times.
[0006] The inference process in LLMs, which involves generating a response based on an input, can vary significantly in duration. This variability is influenced by multiple factors, including the complexity and length of the input query, the specific architecture and size of the LLM, the current computational load on the serving infrastructure (e.g., GPUs, CPUs, memory), I / O bottlenecks, and even the specific content being processed. OS-level challenges such as memory management for large models, CPU scheduling jitter, and I / O handling for model weights and data can further exacerbate these performance inconsistencies.
[0007] This unpredictability poses a considerable problem for systems that rely on timely responses from network functions. Consumer entities (such as other network functions, user equipment, or applications) often operate with predefined timers or timeout mechanisms. If an LLM-NF responds too slowly, the consumer entity might time out prematurely, leading to service failures, degraded user experience, or inefficient resource utilization. Conversely, setting excessively long timers to accommodate worst-case scenarios can lead to unnecessary delays and resource holding.
[0008] SUMMARY
[0009] An objective of the present disclosure is to provide a method, system, and apparatus for estimating the response time of Large Language Model (LLM) based Network Functions (LLM-NFs). A Response Time Estimator Function (RTEF) comprises a network function component configured to calculate and provide an estimated Model Inference Completion Time (MICT) for an LLM-NF before a consumer entity commits to a full service request.
[0010] The foregoing and other objectives are achieved by the features of the independent claims.
[0011] Further implementation forms are apparent from the dependent claims, the description and the Figures.
[0012] A first aspect of the present disclosure provides a method for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times, the method comprising receiving, by a Response Time Estimator Function, RTEF, a request for inference time estimation from a consumer entity in preparation for a service request to an LLM-NF, wherein the RTEFcomprises a network function configured to estimate response times for LLM-NFs, and wherein the consumer entity comprises a network component, user equipment, or network function that requests services from at least one LLM-NF, obtaining, by the RTEF, model information from the LLM-NF, obtaining, by the RTEF, hardware parameters from a Network Management Function, wherein the Network Management Function comprises a network component configured to monitor and provide information about hardware resources hosting the LLM-NF, calculating, by the RTEF, an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time for the LLM-NF to complete processing of the service request, based on at least one of: the obtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits, and providing, by the RTEF, the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for the service request to the LLM-NF.
[0013] By receiving an upfront, calculated estimate of the Model Inference Completion Time (MICT) from a Response Time Estimator Function (RTEF), a consumer entity — be it user equipment or another network function — can make a more informed decision before committing to a full service request. This foresight allows for better management of network resources, as requests to potentially slow or overloaded LLM-NFs can be avoided if the estimated time is unacceptable. Furthermore, it greatly enhances the user experience by preventing unnecessarily long waiting periods or premature timeouts, leading to more predictable and reliable interactions with advanced Al-driven network services.
[0014] In an implementation of the first aspect, the model information can comprise at least one of: model size, model architecture, model complexity, or model processing requirements.
[0015] Incorporating detailed model information, such as the LLM's size, its architectural design, inherent complexity, or specific processing requirements, into the response time estimation method significantly enhances the precision of the predicted MICT. By considering these intrinsic attributes of the language model, the RTEF can produce more reliable and nuanced forecasts of how long an LLM-NF will take to process a request, leading to better-informed decisions by consumer entities and more efficient network operation.
[0016] The hardware parameters can comprise at least one of: GPU specifications, memory capacity, processing power, or current load conditions.
[0017] Factoring in specific hardware parameters, such as the capabilities of the Graphics Processing Units (GPUs), available memory capacity, overall processing power, or current load conditions of the infrastructure hosting the LLM-NF, further refines the accuracy of the MICT estimation. This consideration of the real-time operational context and physical resources allows the RTEF to generate more context-aware and realistic response time predictions, which is crucial for effective service management.
[0018] The method can further comprise receiving, by the RTEF, historical inference time data for the LLM-NF, and wherein calculating the estimated MICT further comprises using the historical inference time data.
[0019] Inclusion of historical inference time data for a particular LLM-NF into the MICT calculation process introduces an adaptive learning capability. By analysing past performance, the RTEF can identify trends, adjust for recurring patterns in processing times, and continuously refine its estimation algorithms. This results in an increasingly robust and accurate prediction mechanism that adapts to the evolving behaviour of the LLM-NF and its operational environment over time.
[0020] The RTEF can be collocated with the LLM-NF. The RTEF can be implemented as a standalone network function.
[0021] Deploying the RTEF in close physical or logical proximity to the LLM-NF it serves, such as being collocated, offers advantages in terms of reduced communication latency when obtaining model-specific information. This close integration can alsosimplify the overall deployment architecture and management, potentially leading to faster MICT estimations and a more streamlined operational flow between the estimation function and the LLM processing unit.
[0022] Implementing the RTEF as a standalone network function provides considerable deployment flexibility and scalability. This architectural choice allows a single RTEF instance to serve multiple LLM-NF s across the network, promoting efficient resource utilization. Furthermore, a standalone RTEF can be managed, updated, and scaled independently of the LLM-NFs, which is particularly beneficial in large, distributed, and dynamically evolving network environments.
[0023] The method can further comprise receiving, by the LLM-NF, the service request from the consumer entity following the inference time estimation, the service request including a maximum response time based on the estimated MICT, and processing, by the LLM-NF, the service request within the maximum response time.
[0024] When the LLM-NF receives a service request that includes a maximum response time derived from the RTEF's MICT estimation, it creates a more accountable service interaction. This explicit communication of performance expectations prompts the LLM-NF to potentially prioritize or manage its internal processing queues and resources more effectively to meet the consumer entity’s timing requirements. This ultimately improves the reliability and predictability of the service delivery once the request has been formally initiated.
[0025] A second aspect of the present disclosure provides a Response Time Estimator Function, RTEF, for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times, wherein the RTEF comprises a network function configured to estimate response times for LLM- NFs, the RTEF comprising, at least one processor, and at least one memory storing instructions that, when executed by the at least one processor, cause the RTEF to receive a request for inference time estimation from a consumer entity in preparation for a service request, wherein the consumer entity comprises a network component, user equipment, or network function that requests services from an LLM- NF, obtain model information from the LLM- NF, obtain hardware parameters from a Network Management Function, wherein the Network Management Function is a network component configured to monitor and provide information about hardware resources hosting the LLM- NF, calculate an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time required for the LLM-NF to complete processing of the service request, based on at least one of: the obtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits, and provide the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for the service request to the LLM- NF.
[0026] The provision of a dedicated network function, the Response Time Estimator Function (RTEF), specifically engineered to forecast response times for LLM-based network functions, offers a specialized and centralized solution to a critical challenge in modern networks.
[0027] In an implementation of the second aspect, the model information can comprise at least one of: model size, model architecture, model complexity, or model processing requirements.
[0028] The hardware parameters can comprise at least one of: GPU specifications, memory capacity, processing power, or current load conditions.
[0029] The instructions can further cause the RTEF to: receive historical inference time data for the LLM- NF, and calculate the estimated MICT further based on the historical inference time data.
[0030] The RTEF can be collocated with the LLM- NF. The RTEF can be implemented as a standalone network function.The LLM- NF can receive the service request from the consumer entity following the inference time estimation, the service request including a maximum response time based on the estimated MICT, and process the service request within the maximum response time.
[0031] When an LLM-NF operates in conjunction with an RTEF and is configured to receive service requests that include a maximum response time based on the MICT, a more predictable service environment is fostered. This configuration ensures that the LLM-NF is explicitly aware of the consumer's timing expectations, encouraging it to manage its processing accordingly and enhancing the overall reliability of the service interaction from the consumer's perspective.
[0032] A third aspect of the present disclosure provides a system for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times, the system comprising a Response Time Estimator Function, RTEF, comprising a network function configured to estimate response times for LLM-NFs, wherein the system is configured to receive a request for inference time estimation from a consumer entity in preparation for a service request, wherein the consumer entity comprises a network component, user equipment, or network function that requests services from an LLM-NF, obtain model information from the LLM-NF, obtain hardware parameters from a Network Management Function, wherein the Network Management Function comprises a network component configured to monitor and provide information about hardware resources hosting the LLM-NF, calculate an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time required for the LLM-NF to complete processing of a service request, based on at least one of: the obtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits, and provide the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for a service request to the LLM-NF.
[0033] A fourth aspect of the present disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for estimating response time in a network comprising Large Language Model, LLM, based network functions, according to the first aspect.
[0034] These and other aspects of the invention will be apparent from the embodiment(s) described below.
[0035] BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order that the present disclosure may be more readily understood, embodiments will now be described, by way of example, with reference to the accompanying drawings, in which:
[0037] Figure 1 is a schematic representation of system, according to an example;
[0038] Figure 2 is a messaging flow for a procedure of inference completion time estimation for signaling timer configuration, according to an example;
[0039] Figure 3 is a schematic representation of a machine according to an example; and
[0040] Figure 4 is as schematic representation of a method, according to an example, for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times.DETAILED DESCRIPTION
[0041] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.
[0042] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.
[0043] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof. The term “and / or” is only an association relationship for describing associated objects and represents that three relationships may exist such that A and / or B may indicate that A exists alone, A and B exist at the same time, or B exists alone. The character “ / ” generally represents that the associated objects are in an “or” relationship. Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.
[0044] The following contains specific information related to implementations of the present disclosure. The drawings and their accompanying detailed disclosure are merely directed to implementations. However, the present disclosure is not limited to these implementations. Other variations and implementations of the present disclosure will be obvious to those skilled in the art.
[0045] The phrases “in one implementation,” or “in some implementations,” may each refer to one or more of the same or different implementations. The term “coupled” is defined as connected whether directly or indirectly through intervening components and is not necessarily limited to physical connections. The expression “at least one of A, B and C” or “at least one of the following: A, B and C” means “only A, or only B, or only C, or any combination of A, B and C.”
[0046] The terms “system” and “network” may be used interchangeably.
[0047] For the purposes of explanation and non-limitation, specific details such as functional entities, techniques, protocols, and standards are set forth for providing an understanding of the present disclosure. In other examples, detailed disclosure of well-known methods, technologies, systems, and architectures are omitted so as not to obscure the present disclosure with unnecessary details.
[0048] Persons skilled in the art will immediately recognize that any network function(s) or algorithm(s) disclosed may be implemented by hardware, software or a combination of software and hardware. Disclosed functions may correspond to modules which may be software, hardware, firmware, or any combination thereof.A software implementation may include machine- and / or computer- readable and / or executable instructions stored on a machine- and / or computer-readable medium such as memory or other types of storage devices. One or more microprocessors or general-purpose computers with communication processing capability may be programmed with corresponding executable instructions and perform the disclosed network function(s) or algorithm(s).
[0049] The microprocessors or general-purpose computers may include Applications Specific Integrated Circuitry (ASIC), programmable logic arrays, and / or using one or more Digital Signal Processor (DSPs). Although some of the disclosed implementations are oriented to software installed and executing on computer hardware, alternative implementations implemented as firmware or as hardware or as a combination of hardware and software are well within the scope of the present disclosure. The computer readable medium includes but is not limited to Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, Compact Disc Read-Only Memory (CD-ROM), magnetic cassettes, magnetic tape, magnetic disk storage, or any other equivalent medium capable of storing computer-readable instructions.
[0050] According to an example, a Response Time Estimator Function (RTEF) comprises a component implemented as a standalone service, a library integrated into other services, or a hardware module.
[0051] For example, an agentic NF can be enhanced with a RTEF component. With such an enhancement, a consumer entity first interacts with the RTEF module to get an estimated MICT before formally sending a service request. The MICT will be derived by RTEF by considering the input information (i.e., the prompt), the model information (including its structure and sequence limit) and the hardware information where the agentic NF producer with the LLM is deployed. With the MICT information, the consumer entity will further decide its following signaling actions (continue or terminate if necessary).
[0052] Figure 1 is a schematic representation of system, according to an example. In the example of figure 1, a Consumer Entity (UE / NF) 101 comprises an initiator of service requests. The consumer entity can be an end-user device or another function within a network that requires services from the Agentic NF 103, which, according to an example, comprises an LLM-powered NF responsible for providing services. The agentic NF 103 is, in the example of figure 1, enhanced with the RTEF 105. The RTEF 105 is configured to estimate the response time (MICT) of the Agentic NF for a potential service request. The RTEF can also be deployed as a standalone function.
[0053]
[0054] Other agentic NF functional modules 109 can be provided. These can comprise other operational modules within the Agentic NF 103 that can contribute to its overall service delivery, distinct from the LLM processing itself but part of the agentic behaviour.
[0055] In an example, a Network Resource Manager Function can be provided for managing and providing information about the underlying hardware resources where the agentic NF producer with the LLM is deployed. That is, it can provide the RTEF 105 with details about, e.g., the GPU (Graphics Processing Unit) specifications and utilization, which are used for accurate response time estimation.
[0056] Figure 2 is a messaging flow for a procedure of inference completion time estimation for signaling timer configuration, according to an example.With reference to figure 2, the following new services provided by RTEF 105 and other relevant components can be provided:
[0057]
[0058] Referring to figure 2, an Initial MICT Request (1, referred to as ‘Nrtef lnferenceTimeEstRequest’) is sent from the Consumer Entity 101 to the RTEF 105. This request typically includes parameters related to the intended service, such as the LLM-NF ID where the service request will be handled, service name (or input prompt characteristics), maximum output sequence length, and potentially a preferred estimation method.
[0059] As noted above, consumer entity 101 can be either a UE or an NF in a core wireless mobile telecommunications network. If the consumer entity is a UE, the service request and the following interactions can occur over, e.g., the network access stratum (NAS). If the consumer entity is an NF, the service request can be sent using a service-based interface in the core network. Upon receiving the MICT request, the RTEF 105 queries (2a) the LLM-NF 107 (‘Nllmnf ModellnfoRequest’) to retrieve essential parameters about the LLM itself. This includes information like model size, architecture, number of layers, and sequence length limits. This request aims to retrieve the parameters of the structure of the LLM-NF according to the LLM-NF ID.
[0060] Concurrently or sequentially, the RTEF 105 sends (2b) a request (e.g., ‘NnwMgmtf GPUParamRequest’) to the Network Resource Manager Function 201. This request is to obtain information about the hardware resources, such as the GPU specifications (e.g., TFLOPs, precision like FP16 / FP32) on which the LLM-NF 107 is deployed and currently operating. Both the LLM-NF 107 and the Network Resource Manager Function 210 respond (3a, 3b) to the RTEF 105, providing the requested model parameters and hardware specifications, respectively.With all the necessary input (prompt characteristics from consumer, model details from LLM-NF, and hardware details from Network Resource Manager), the RTEF 105 calculates (4) the estimated MICT.
[0061] The RTEF 105 sends (5) the calculated MICT back to the Consumer Entity 101. This response includes the estimated MICT, the service ID, and the LLM-NF ID. The Consumer Entity 101 can then evaluate the received MICT. If the estimated time is acceptable for the intended service, the consumer can set a local timer (6) for the subsequent service request with a value based on (e.g., equal to) the MICT. If the MICT is unacceptable, the consumer may decide to terminate the process and not send the service request.
[0062] If the MICT is acceptable, the Consumer Entity 101 can proceed to send (7) the actual service request to the Agentic NF (specifically to the LLM-NF 107 or relevant functional modules 109). This request includes the service ID and the maximum response time (based on MICT) that the Agentic NF 103 should be aware of. The service request can be sent over the Nagnf ServiceRequest interface.
[0063] After processing the request, the Agentic NF 103 sends (8) the service response back to the Consumer Entity 101.
[0064] According to an example, to estimate the LLM inference completion time given an input sequence and an output sequence limit (i.e., sequence length limit), the following factors influence the processing time:
[0065] • Input Sequence Length: The number of tokens in the input sequence
[0066] • Output Sequence Limit: The maximum number of tokens the model will generate for the output
[0067] • Model Architecture and Parameters: The size of the model and the number of layers in the model
[0068] • GPU Specifications: The computational capability of the GPU (e.g., NVIDIA A100 with 156 TFLOPs for FP16).
[0069] Precision: Whether the inference is running in FP32, FP16, or INT8 (lower precision models are faster).
[0070] In an example, inference completion time can be estimated using:
[0071] FLOPs per inference
[0072] Inference Time (seconds = - ; - - - '7GPU FlOPs / sec
[0073] For example, with a set of input parameters:
[0074] • Model: GPT-3 (175B parameters)
[0075] • Sequence Length: 512 tokens (input)
[0076] • Output Sequence Limit: 150 tokens (max output)
[0077] • Number of Layers: 96 layers (for GPT-3)
[0078] • GPU: NVIDIA A100 (156 TFLOPs, FP16)
[0079] The FLOPs per inference = 2 x 175 x 109x 96 x (512 + 150) = 2.1 x 1013FLOPS
[0080] Where the factor of 2 accounts for the two computations needed in the self-attention mechanism of transformers (attention score calculation and weighting).
[0081] The number of GPU FLOPs / sec= 156 x 1012FLOPs / sec
[0082] Combining the two parts, the total inference completion time is:Inference Time (seconds)= — - — « 0.1346 seconds
[0083] This example shows that if the LLM-NF 107 is empowered with, e.g., GPT-3, given an input prompt with 512 tokens and allowed to output with 150 output tokens and hosted with an, e.g., NVIDIA A100 GPU, such a request will roughly take 0.13s to complete the inference. This value will be the value of MICT returned to the consumer entity for further decision.
[0084] According to an example, RTEF 105 can estimate an MICT, based on a set of received parameters, using, any one or more of e.g., historical data to build statistical models that predict response time based on current conditions, machine learning models (e.g., a regression model or a neural network) to predict response time based on various input features. The choice will depend on the specific requirements of the application, such as the desired accuracy, computational resources available, and the complexity of the LLM-NF.
[0085] The RTEF 105 output, which comprises an estimated response time for the LLM-NF 107, can comprise a single value (e.g., average response time) or a range of values (e.g., 90% confidence interval).
[0086] As described above, the RTEF can be implemented as a standalone network service or microservice. In such a scenario, the RTEF service can be accessible via an API (e.g., over HTTP / S or a gRPC interface) by any authorized Consumer Entity within the network.
[0087] The standalone RTEF service can include a Data Collection Module, a Data Storage Module (potentially a dedicated timeseries database for performance data), a Prediction Engine, and an API Interface. It can be deployed on its own set of virtual or physical resources.
[0088] For a standalone RTEF service, a Consumer Entity (e.g., an application server, another NF) intending to use an LLM-NF can, as described above with reference to figure 2 for example, send an estimation request to the standalone RTEF service. The request can include an identifier for the target LLM-NF and potentially parameters about the intended service request (e.g., expected input length, task type).
[0089] The RTEF service queries the target LLM-NF (via its management interface or a dedicated API) to obtain current model information (e.g., model version, loaded parameters).
[0090] The RTEF service queries the Network Management Function (which could be a central orchestrator or a local monitoring agent) to obtain hardware parameters for the infrastructure hosting the target LLM-NF (e.g., GPU utilization, available memory).
[0091] The RTEF’s Prediction Engine uses these inputs, perhaps along with, e.g., historical performance data for that LLM-NF (retrieved from its Data Storage Module), to calculate the MICT. The RTEF service returns the MICT to the Consumer Entity. This enables a centralized approach for easier management, updating, and scaling of the RTEF service itself. It can serve multiple LLM-NFs and Consumer Entities. Historical data from various LLM-NFs can be aggregated to improve the prediction models over time.
[0092] In another example, the RTEF functionality can be implemented as a component or module that is collocated or tightly integrated with each LLM-NF. That is, the RTEF module can be a sidecar container in a microservices deployment (e.g., a Kubernetes pod alongside the LLM-NF container) or a library directly linked into the LLM-NF software.In such a scenario, a Consumer Entity can send an estimation request, potentially directly to a specific management endpoint of the LLM-NF that exposes the collocated RTEF’s functionality. The collocated RTEF module can access the LLM-NF’s model information locally and query the Network Management Function for hardware parameters (this can be a remote call, or the NMF agent could also be collocated).
[0093] The Prediction Engine, running within the collocated RTEF module, calculates the MICT. It might use a model specific to this LLM-NF instance or a more general model configured for it. The MICT is returned to the Consumer Entity, possibly via the LLM-NF’ s management interface.
[0094] There is therefore reduced latency for obtaining model information from the LLM-NF, and the RTEF can be tailored specifically to the characteristics of the LLM-NF it is paired with.
[0095] In another example, a Prediction Engine within the RTEF, irrespective of whether the RTEF is standalone or collocated, can use an ML model. Examples include:
[0096] • Regression Models: Linear regression, polynomial regression, gradient boosting regressors (e.g., XGBoost, LightGBM), or support vector regression (SVR) could be trained to predict MICT based on features.
[0097] • Neural Networks: A feedforward neural network or a recurrent neural network (if sequence of operations matters) could be trained for more complex pattern recognition.
[0098] Input features for the ML model could include:
[0099] • Model Information: Encoded model type, model size (parameters), embedding dimensions, number of layers.
[0100] • Hardware Parameters: GPU clock speed, GPU memory available, CPU cores, CPU utilization, network bandwidth.
[0101] • Request-Specific Parameters: Input sequence length, output sequence length (if predictable or limited), query complexity score (e.g., derived from text analysis of the prompt), batch size (if applicable). • Historical Performance: Rolling averages of past MICTs for similar requests, recent error rates.
[0102] The ML model can be trained offline using a historical dataset of LLM-NF interactions and their actual completion times. The model can be periodically retrained (e.g., daily or weekly) with new data to adapt to evolving performance characteristics or new LLM-NF versions.
[0103] Thus, complex, non-linear relationships between various factors and the MICT can be captured, potentially leading to more accurate estimations than simpler statistical models, especially in highly dynamic environments.
[0104] In another example, the RTEF can be integrated with a network orchestrator or a service management and orchestration (SMO) framework, which can be relevant in telecommunications contexts (e.g., 5G / 6G networks). The RTEF can therefore provide its MICT estimates not only to Consumer Entities but also to the orchestrator. The orchestrator can use this information for resource allocation, service chaining, or admission control decisions.
[0105] For example, a Consumer Entity can request a service that involves an LLM-NF. Before instantiating or routing to an LLM-NF, the orchestrator can query the RTEF for an estimated MICT for the required task on available LLM-NF instances. Based on the MICT and current network policies (e.g., SLOs for the service), the orchestrator can, e.g., select the most suitable LLM-NF instance (e.g., one that can meet the latency requirement), allocate additional resources to an LLM-NF if the estimated MICT is too high with current resources, or deny the request if no LLM-NF can meet the required MICT within policy limits. The orchestrator can then inform the Consumer Entity of the decision and / or the selected LLM-NF and its expected MICT. Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.
[0106] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.
[0107] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, an RTEF or other NF or NFs, a platform comprising user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus (for example, a module implementing a Data Collection Module, a Data Storage Module, a Prediction Engine) may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors.
[0108] Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non-transitory computer readable storage medium encoded with instructions, executable by a processor.
[0109] Figure 3 is a schematic representation of a machine according to an example. The machine 300 can be, e.g., a system or apparatus, an RTEF, user equipment, or part thereof. The machine 300 comprises a processor 303, and a memory 305 to store instructions 302, executable by the processor 303. The machine comprises a storage 309 that can be used to store data 301 representing matrices and / or the values of their elements, scaling factors and so on as described above with reference to figure 1 for example.
[0110] The instructions 307, executable by the processor 303, can cause the machine 300 to receive a request for inference time estimation from a consumer entity in preparation for a service request to an LLM-NF, wherein the machine can comprise, e.g., an RTEF comprising a network function configured to estimate response times for LLM-NFs, and wherein the consumer entity comprises a network component, user equipment, or network function that requests services from at least one LLM-NF, obtain model information from the LLM-NF, obtain hardware parameters from a Network Management Function, wherein the Network Management Function comprises a network component configured to monitor and provide information about hardware resources hosting the LLM-NF, calculate an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time for the LLM-NF to complete processing of the service request, based on at least one of: theobtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits, and provide the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for the service request to the LLM-NF.
[0111] Accordingly, the machine 300 can implement a method for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times.
[0112] Such machine-readable instructions may also be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.
[0113] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure.
[0114] In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloud-computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.
[0115] Figure 4 is as schematic representation of a method, according to an example, for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times.
[0116] In block 401, a Response Time Estimator Function, RTEF, receives a request for inference time estimation from a consumer entity in preparation for a service request to an LLM-NF, wherein the RTEF comprises a network function configured to estimate response times for LLM-NFs, and wherein the consumer entity comprises a network component, user equipment, or network function that requests services from at least one LLM-NF.
[0117] In block 403, the RTEF obtains model information from the LLM-NF. In block 405, the RTEF obtains hardware parameters from a Network Management Function, wherein the Network Management Function comprises a network component configured to monitor and provide information about hardware resources hosting the LLM-NF.
[0118] In block 407, the RTEF calculates an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time for the LLM-NF to complete processing of the service request, based on at least one of: the obtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits.
[0119] In block 409, the RTEF provides the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for the service request to the LLM-NF.
[0120] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of theparticular type of computer-readable-storage media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer- readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein. In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another.
[0121] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.
Claims
CLAIMS1. A method for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times, the method comprising:receiving, by a Response Time Estimator Function, RTEF, a request for inference time estimation from a consumer entity in preparation for a service request to an LLM-NF, wherein the RTEF comprises a network function configured to estimate response times for LLM-NFs, and wherein the consumer entity comprises a network component, user equipment, or network function that requests services from at least one LLM-NF;obtaining, by the RTEF, model information from the LLM-NF ;obtaining, by the RTEF, hardware parameters from a Network Management Function, wherein the Network Management Function comprises a network component configured to monitor and provide information about hardware resources hosting the LLM-NF;calculating, by the RTEF, an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time for the LLM-NF to complete processing of the service request, based on at least one of: the obtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits; andproviding, by the RTEF, the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for the service request to the LLM-NF.
2. The method of claim 1, wherein the model information comprises at least one of: model size, model architecture, model complexity, or model processing requirements.
3. The method of claim 1, wherein the hardware parameters comprise at least one of: GPU specifications, memory capacity, processing power, or current load conditions.
4. The method of any preceding claim, further comprising:receiving, by the RTEF, historical inference time data for the LLM-NF, and wherein calculating the estimated MICT further comprises using the historical inference time data.
5. The method of any preceding claim, wherein the RTEF is collocated with the LLM-NF.
6. The method of any preceding claim, wherein the RTEF is implemented as a standalone network function.
7. The method of any preceding claim, further comprising:receiving, by the LLM-NF, the service request from the consumer entity following the inference time estimation, the service request including a maximum response time based on the estimated MICT; andprocessing, by the LLM-NF, the service request within the maximum response time.
8. A Response Time Estimator Function, RTEF, for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NF s, utilizing at least one LLM to process service requests with variable and non-deterministic response times, wherein the RTEF comprises a network function configured to estimate response times for LLM- NFs, the RTEF comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the RTEF to: receive a request for inference time estimation from a consumer entity in preparation for a service request, wherein the consumer entity comprises a network component, user equipment, or network function that requests services from an LLM-NF;obtain model information from the LLM- NF;obtain hardware parameters from a Network Management Function, wherein the Network Management Function is a network component configured to monitor and provide information about hardware resources hosting the LLM- NF; calculate an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time required for the LLM- NF to complete processing of the service request, based on at least one of: the obtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits; andprovide the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for the service request to the LLM- NF.
9. The RTEF of claim 8, wherein the model information comprises at least one of: model size, model architecture, model complexity, or model processing requirements.
10. The RTEF of claim 8 or 9, wherein the hardware parameters comprise at least one of: GPU specifications, memory capacity, processing power, or current load conditions.
11. The RTEF of any of claims 8 to 10, wherein the instructions further cause the RTEF to: receive historical inference time data for the LLM- NF; and calculate the estimated MICT further based on the historical inference time data.
12. The RTEF of any of claims 8 to 11, wherein the RTEF is collocated with the LLM- NF.
13. The RTEF of any of claims 8 to 11, wherein the RTEF is implemented as a standalone network function.
14. The RTEF of any of claims 8 to 13, wherein the LLM- NF is configured to:receive the service request from the consumer entity following the inference time estimation, the service request including a maximum response time based on the estimated MICT; andprocess the service request within the maximum response time.
15. A system for estimating response time in a network comprising Large Language Model, LLM, based network functions, LLM-NFs, utilizing at least one LLM to process service requests with variable and non-deterministic response times, the system comprising a Response Time Estimator Function, RTEF, comprising a network function configured to estimate response times for LLM-NFs, wherein the system is configured to:receive a request for inference time estimation from a consumer entity in preparation for a service request, wherein the consumer entity comprises a network component, user equipment, or network function that requests services from an LLM-NF;obtain model information from the LLM-NF;obtain hardware parameters from a Network Management Function, wherein the Network Management Function comprises a network component configured to monitor and provide information about hardware resources hosting the LLM-NF;calculate an estimated Model Inference Completion Time, MICT, wherein the MICT comprises an estimated time required for the LLM-NF to complete processing of a service request, based on at least one of: the obtained model information, the hardware parameters, the content of the service request, input sequence length limits, output sequence length limits; and provide the estimated MICT to the consumer entity to enable the consumer entity to set an appropriate timer for a service request to the LLM-NF.
16. A non-transitory computer- readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for estimating response time in a network comprising Large Language Model, LLM, based network functions, according to the method of any of claims 1 to 7.16
