Method for training a large language model (LLM) to detect events in a time series

Abstract

The disclosure relates to a method, and apparatus, for training a large language model (LLM) to detect events in a time series. The method comprises locating an event in a time series training dataset using a knowledge corpus. The method comprises generating, from the time series, time window samples having varying lengths and starting points, the time window samples including the event. The method comprises extracting time series data, associated with the event, within the time window samples. The method comprises training the LLM to detect events in the time series, using the event and the extracted time series data.

Description

P112430W0011METHOD FOR TRAINING A LARGE LANGUAGE MODEL (LLM) TO DETECT EVENTS IN A TIME SERIESTECHNICAL FIELD

[0001] The present disclosure relates to a smart data sampling solution for building an anomaly detection model using pre-trained large language models for network operation.BACKGROUND

[0002] With the rapid evolution of digital technology, physical networks owned and operated by different telecom service provider connect not only phones but also devices, which become more and more intelligent. These devices share the information and services with each other.

[0003] The physical networks operate through a combination of hardware (like routers, switches, and cables) and protocols (rules for data transfer) that allow communication between connected devices as well as applications deployed in the cloud.

[0004] Here is the basic breakdown of how the networks operate:- Data transmission between devices.- Routing and switching for transferring the data packet.- Protocols for data transmission, e.g., transmission control protocol (TCP) / intemet protocol (IP), hypertext transfer protocol (HTTP), etc.- Wired and wireless transmission.Client-Server model.Security.- Network layers, open system interconnection (OSI) model.

[0005] To make networks run properly without any problem / incidence, such as service interruption, etc., an operation and maintenance (0AM) service is mandatory. Traditionally, in the operator’s network, the services of the 0 AM are performed by its network engineers using the monitoring tools (including both software and hardware) offered by the equipment vendors, inhouse developers, or external third-party.P112430W0012

[0006] Looking at current network deployments, one of the major operations, i.e., to detect anomalies in the network, mainly relies on network engineers who do manual operations using their expertise (knowledge and experience).

[0007] In general, those network engineers not only look at a dashboard that displays key performance indicator (KPIs), counters, alarms, and notifications, but also retrieve information from different logs generated in the network to do the required analysis. After collecting all the information successfully, they still need a lot of time (days or weeks) to figure out a root cause if any incident that occurred. It should also be pointed out that not all incidents occurring in the network can be resolved successfully due to the task complexity. This leads to extremely high cost for network operation, in other words high operating expense (OPEX).SUMMARY

[0008] There is provided a computer implemented method for training a large language model (LLM) to detect events in a time series. The method comprises locating an event in a time series training dataset using a knowledge corpus. The method comprises generating, from the time series, time window samples having varying lengths and starting points, the time window samples including the event. The method comprises extracting time series data, associated with the event, within the time window samples. The method comprises training the LLM to detect events in the time series, using the event and the extracted time series data.

[0009] There is provided an apparatus operative to train a large language model (LLM) to detect events in a time series. The apparatus comprises processing circuits and a memory. The memory contains instructions executable by the processing circuits whereby the apparatus is operative to locate an event in a time series training dataset using a knowledge corpus. The apparatus is operative to generate, from the time series, time window samples having varying lengths and starting points, the time window samples including the event. The apparatus is operative to extract time series data, associated with the event, within the time window samples. The apparatus is operative to train the LLM to detect events in the time series, using the event and the extracted time series data.

[0010] There is provided a non-transitory computer readable media having stored thereon instructions for training a large language model (LLM) to detect events in aP112430W0013time series. The instructions comprise locating an event in a time series training dataset using a knowledge corpus. The instructions comprise generating, from the time series, time window samples having varying lengths and starting points, the time window samples including the event. The instructions comprise extracting time series data, associated with the event, within the time window samples. The instructions comprise training the LLM to detect events in the time series, using the event and the extracted time series data.

[0011] The method and apparatus provided herein present improvements to the way anomaly detection or detection of events in a timeseries operate.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic illustration comparing a proposed data sampling method with a traditional data sampling method.

[0013] Figure 2 is a flowchart illustrating the smart slicing time window mechanism.

[0014] Figure 3 is a schematic illustration showing an example data sampling with multiple events.

[0015] Figure 4 is a block diagram showing a component view of the proposed solution in the model training process.

[0016] Figure 5 is a flowchart illustrating activities of the event-based data sampling module.

[0017] Figure 6 is a flowchart illustrating activities of the knowledge-based data sampling module.

[0018] Figure 7 is a block diagram showing a component view of the proposed solution in the model inference process.

[0019] Figure 8 is a flowchart illustrating steps of the anomaly detection for time series data module.

[0020] Figure 9 is a block diagram showing an overview of log anomaly detection engine with fine-tuned Llama 3 distributed learner.

[0021] Figure 10 is a schematic illustration of the data sampling and preprocessing engine.

[0022] Figure 11 illustrates an example message log from a teread.json file, which indicates there is an error / failure.

[0023] Figure 12 illustrates two example anomaly cases with text messages.P112430W0014

[0024] Figure 13 illustrates model performance on log failure detection with three different test cases.

[0025] Figure 14 is a flowchart of a method for training a large language model (LLM) to detect events in a time series.

[0026] Figure 15 is a schematic illustration of a hardware in which steps and / or method described herein can be executed.

[0027] Figure 16 is a schematic illustration of a virtualization environment in which the different steps and hardware components described herein can be deployed.DETAILED DESCRIPTION

[0028] Various features will now be described with reference to the drawings to fully convey the scope of the disclosure to those skilled in the art.

[0029] Sequences of actions or functions may be used within this disclosure. It should be recognized that some functions or actions, in some contexts, could be performed by specialized circuits, by program instructions being executed by one or more processors, or by a combination of both.

[0030] Further, computer readable carrier or carrier wave may contain an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein.

[0031] The functions / actions described herein may occur out of the order noted in the sequence of actions or simultaneously. Furthermore, in some illustrations, some blocks, functions, or actions may be optional and may or may not be executed; these are generally illustrated with dashed lines.

[0032] At least some aspects of the techniques described herein may be implemented using artificial intelligence, which comprises a variety of techniques as would be apparent to a person skilled in the art, including machine learning techniques.Machine learning techniques include deep learning and neural network (NN), or artificial neural network (ANN). Both NN and ANN may be used interchangeably herein. In some contexts, an Artificial Neural Network could include biological portions.

[0033] Further, looking forward, in a virtual world (e.g., the metaverse, digital twins, etc.), the techniques described herein could be applied in relevant virtual scenarios.P112430W0015

[0034] As stated in the background, network engineers look at information collected in the network to analyse incidents that occurred in the network.

[0035] Most of the information collected in the network, e.g., KPIs, counters, alarms, log files, etc., contain messages written in human language to facilitate the analysis done by the network engineers. Hence one solution to reduce OPEX could be to use a Large Language Model (LLM) to assist in this field.

[0036] Reference will be made to the following reference documents:(1) Large Language Models: A Survey. S. Mlnaee, T. Mikolov, N. Nikzad, M.Chenaghlu, R. Socher, X. Amatrain, J. Gao, v2, submitted on 20 Feb 2024, IEEE, Computer Science / Computation and Language.(2) A Survey of Large Language Models. W.X. Zhao, K. Zhou, J. Li, T. Tang, et al. vl5 submitted on 13 Oct 2024, IEEE Computer Science / Computation and Language. (3) Efficient Estimation of Word Representations in Vector Space. T. Mikolov, K. Chen, G. Corrado, J. Dean, v3, submitted on 7 Sept 2013, IEEE Computer Science / Computation and Language.(4) GloVe: Global Vectors for Word Representation. J. Pennington, R. Socher, C. D. Manning, vl, submitted on Jan 2014, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).(5) Attention is all you need. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin, v7, submitted on 2 August 2023 IEEE Computer Science / Computation and Language.(6) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. J. Devlin, M. Chang, K. Lee, K. Toutanova, v2, submitted on 24 May 2019 IEEE Computer Science / Computation and Language.(7) Language Models are Few-Shot Learners. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, et al. v4. Submitted on 22 July 2020, IEEE Computer Science / Computation and Language.

[0037] Evolution of LLMs and their capabilities is discussed next.

[0038] The evolution of LLMs has transformed the landscape of natural language processing (NLP) over the past decade (1 )(2). Initially, NLP relied on rule-based systems and simpler machine learning techniques, which struggled to capture the complexity of human language. The introduction of deep learning revolutionized theP112430W0016field, with models like Word2Vec (3) and Global Vectors for Word Representation (GloVe) (4) providing better word embeddings that captured semantic relationships.

[0039] The release of the transformer architecture in 2017 (5) marked a pivotal moment. Transformers enabled models to process words in relation to all other words in a sentence simultaneously, enhancing context understanding. This led to the development of groundbreaking models like BERT (6) and GPT (7), which utilized large-scale datasets for pre-training and fine-tuning.

[0040] As research progressed, LLMs grew in scale and capability. Models like GPT-3 / GPT-4 showcased the ability to generate coherent, contextually relevant text across various topics, while innovations in architecture and training techniques have improved efficiency and performance.

[0041] Today, LLMs are not only tools for text generation but also for understanding and interacting with human language in a more nuanced way, paving the path for applications in chatbots, content creation, and beyond.

[0042] Challenges to reduce OPEX for network operator are discussed next.

[0043] As stated in previously, 0AM services are provided by experienced network engineers. The detection of anomaly events occurring in the network mainly relies on those network engineers, who look through massive data collected through counters, logs and KPIs etc. The cost associated to this kind of operations is high due to the nature of the human involvement as well as the complexity of the network in which the incidents occur.

[0044] In addition, the networks have been evolving towards more intelligent and sophisticated networks. Nowadays, for instance, fifth generation (5G) networks are much more complicated than the previous mobile networks generations (2G / 3G / 4G). Much more network engineers are required to work together to make sure that the network runs smoothly and efficiently without any service interruption or downtime.

[0045] One big challenge for all the network operators is how to keep OPEX under control while the mobile networks further evolve to have more complex features (such as network slicing in 5G). Automation and Optimization in the process of 0AM services are therefore needed.

[0046] Further, with the rapid development of the telecom technology, the deployment of nodes in operator’s network also becomes very complicated. For instance, the co-existing deployment of 3G, 4G and 5G radio units as well as the connection among them adds a lot of complexity. In such a case, even for theP112430W0017experienced network engineers, it might take hours, days, or weeks to figure out the root cause for an incident, and eventually find / identify a solution to fix it.

[0047] There are even more difficult issues / tasks to face for those experienced network engineers if they are asked to try to detect an anomaly ahead of the potential upcoming incidents.

[0048] Given these challenges, a solution using LLM based anomaly detection is proposed herein, to replace the “experienced network engineers”.

[0049] To build a reliable model efficiently, a pre-trained LLM is taken as the foundation model; and is adapted to become telecom domain specific, e.g., it is adapted to radio unit / antenna, using a supervised machine learning ML technique.

[0050] To achieve a model accuracy level that can be trusted for engineering applications through this supervised ML training process, a large, labeled dataset is needed, which can be taken from the field or the lab.

[0051] Usually, good data samples are easy to obtain from the field, but erroneous data samples are not so easy to obtain. Therefore, there is often a need to generate synthetic erroneous datasets for time-series log analysis.

[0052] Automatically labeling a large amount of data that is collected from the field is also an important element to succeed to train the ML model in a supervised fashion.

[0053] A mechanism to generate a large amount of labelled data samples from limited time series can be described in the following two steps:1. Locate special events in time series dataset using the knowledge corpus, which is built by the network experienced engineers.2. Generate the data samples / frames by varying the length of the slice window around the special events to:a. capture the activities around the special events, e.g., alarms or notifications,b. capture the trends or accumulation of the activities before the special events, andc. capture the trends or accumulation of the activities after the special events.

[0054] In this way, the hidden patterns that link to the special events can be learned by a large LLM model. At the same time these special events are used to label the dataset.P112430W0018

[0055] A special event is identified according to the definition given in a domain specific corpus in the format of a list of key words or one or more paragraphs. The domain specific corpus can be enhanced via the experts in the field (e.g., telecom, aerospace, health care, etc.) or an AI / ML based expert system, such as a LLM based Al agent that learns the knowledge extracted from the monitored system automatically.

[0056] Figure 1 illustrates the proposed solution (right side of the figure), which samples data around the event of interest. Then, this target event is used as a label for the data collected using the new mechanism proposed herein. A traditional time series dataset sampling is also provided for comparison (left side of the figure).

[0057] Referring to the left part in Figure 1, the traditional method of sampling timeseries data is presented. Here, a fixed time slicing window is used. The length of the slicing window can be described as 8t= tk-= tfc+1- tk. All the data within this time window is put together as a single data sample. Then a check can be performed to see if an event 10 can be found within the time window. If the event 10 is found, this data sample is labelled as “positive - anomaly”, otherwise it is labelled as “negative - normal”. The number of labelled data sampled as “positive” are limited by the number of the events occurring in a monitored period. This leads to an unbalance of data samples among which there is a big bias towards “negative” because the network runs successfully most of time.

[0058] A new approach to sample the data points is given in the right part of Figure 1. It is an event focus approach. The proposed method can create a flexible slicing time window around the event to collect the data, to generate more data samples from the collected data when an event is detected. Those generated data samples are used to fine-tune the pre-trained LLM. In the figure, three examples are given:1. The event occurs closer to the beginning of the time window.2. The event occurs in the middle of the time window.3. The event occurs closer to the end of the time window.

[0059] It should be noted that the length of the time window can vary from one window to the other. This is controlled by the slicing window policy. One advantage of the proposed solution over the traditional one is to have much more labelled data points that leads to a balanced dataset between “Positive” and “Negative” data samples, which is a necessary condition to train the supervised large ML model successfully.P112430W0019

[0060] The definition of the slicing window policy includes two different options: one is for model training and the other is for model inference. The elements of these definitions for the slicing window policy are listed hereafter respectively.

[0061] For model training:1. Event-ID.2. Target time period for sampling the data (star time, time length).3. The length of the slicing window (< time length of the target time period). a. The range of window size.i. Time length before eventii. Time length after eventb. Option for window size selection.i. Constantii. Random (default)iii. Round Robiniv. Customized option for extension4. The number of labelled samples generated around events.5. The address of the domain specific corpus (DSC) where events are defined.6. A list of addresses to the raw data resources.

[0062] For model inference:1. Target time period for sampling the data.a. (star time, time length)b. No-stopc. end time2. The length of slicing window (< time length of the target time period).a. The range of window size.i. Time lengthb. Option for window size selection.i. Constant3. The time range to identify the event (e.g. 1 min, 5 mins, etc.).4. A list of addresses to the raw data resources.

[0063] The following example demonstrates how the flexible slicing time window can be calculated.

[0064] In the example, the following is assumed:1. Time length before event for the window is 10 mins.P112430W001102. Time length after event for the window is 30 mins.3. Selection method: Random.4. The number of labelled samples around events: 10.5. The event: “notification on 90% CPU usage”.

[0065] According to the event definition, the system can locate this event from logs if similar message exists, for instance, it occurs at 10:45am on Feb 20, 2024.

[0066] The system can then calculate the total time window for sampling around the event as:1. Event_start = 10:45 - 10 = 10:35.2. Event end = 10:45 + 30 = 11:15.3. Slicing time window = Rand (seed)*( 10+30), where 0 < Rand(seed) < 1. a. If Rand(seed) = 0.3, slicing time window = 12 mins.b. If Rand(seed) = 0.7, slicing time window = 28 mins.

[0067] As for the starting time of the slicing time window, the same Rand (seed) can be used to compute it or another Rand (seed) can be used.c. If the same Rand(seed) = 0.3 is used, slicing time window start 0.3x10=3 mins before Event start: 10:45 -3 min = 10:42. d. If a different Rand(seed), e.g., Rand(seed) = 0.2 is used, slicing time window start 0.2x10=2 mins before Event_start: 10:45 -2 min = 10:43.

[0068] Figure 2 illustrates the activity diagram of the “smart data sampling mechanism” using the mechanism and the slicing time window described above.

[0069] The slicing window policy provides the rules to create a large amount of labelled data points, which are required to train the domain specific ML model based on the pre-trained LLM model.

[0070] The general steps for the smart slicing time window mechanism include:1. Select one event and load the slicing window policy for this event.2. Calculate the slicing time window according to the slicing window policy as well as the slicing time window start time.3. Retrieve all the data points within the slicing time window.4. Concatenate all the messages retrieved from those data points respecting the chronological order (timeline). This is done to catch the trends (hidden pattern) before and after the event.5. Verify if the number of the required data samples around the event is met or not.P112430W00111a. If no, go back to step 2.b. If yes, break and go to end.

[0071] For clarification, an example of data sampling with multiple events is given in Figure 3. It can be seen in the example of figure 3 that events B and C have two time separated data samples each. For example, event B consists of an event where a CPU reaches its limit. This does not happen suddenly as it is an event that generally builds up over time. Data sample 10a can be seen as a precursor of the event that happens during data sample 10b. By collecting both data samples 10a and 10b, it is possible to see an event building up and learning this pattern. Those events might be explicitly correlated as explained above, but they can also be implicitly linked. It should be noted that events A, B, C can be defined as a normal or regular events on their own and not necessary “disaster” events in the traditional terminology in telecom operation services. In some instances, it is the sequence of events that is indicative of a problematic event.

[0072] The advantages of the proposed solution include:Significant reduction on OPEX for network operator.- Detection of anomaly incidents, which is hard even for the experience network engineers.- Automatic knowledge enhancement for domain specific corpus.- Paving the way to detect potential anomaly activities ahead of time to allow reacting in a proactive way rather than in a reactive way.

[0073] Data sampling for model training is described next.

[0074] Figure 4 illustrates a solution for sampling data from raw data sources. A live system 401, e.g. telecom network (4G / 5G / 6G), is under surveillance. One central storage 402, or distributed storages, are provided to collect the alarms and notifications due to occurrences of special events in the live system. In addition, the system also produces logs, counters and KPIs, which are also kept in the storage 402 as shown in Figure 4. This solution can be easily integrated into the network management system to retrieve the related information to create a large amount of data samples for training the anomaly detection model.

[0075] The proposed solution has six components:Special event receivers (SER) 403 : this component subscribes to the alarms and notifications for the network nodes deployed in the operator network.P112430W00112- Event based data sampling module (EBDSM) 404: this component is invoked by the events occurring in the network. The time when an event occurred is used as a reference point to trigger the collection of the relative information from different raw data resources according to the slice window policy. Then the slicing time window is calculated, and the data is generated with the event as a label. EBDSM 404 stores all the generated data samples around the event into the storage 402 for training the model later.- Data storage 402: this is where the generated labelled data points are stored. - Domain specific corpus (DSC) 405: this component contains the domain knowledge and experience built by experienced network engineers or AI / ML based model.- Knowledge based data sampling module (KBDSM) 406: this component is designed to utilize the DSC to locate the events that occurred in the dataset retrieved from different raw data sources. According to the event definition, this component is able to locate different events from the time series dataset collected from the field, e.g. the nodes deployed in the telecom operator network. Then it automatically generates the data samples around those events and stores them into the database or data storage. Those data samples are labelled per event and will be used to train the event related model accordingly.LLM based event retrieval module (LLM-BERM) 407: this component fetches a batch of labelled (event related) points and puts them together as one or multiple paragraphs. Then the pre-trained LLM based model (e.g., ChatGPT) is used to extract the key words from those paragraphs. Those key words describe the event, which can be put into the DSC 405.

[0076] The business logics in EBDSM module 404 is presented in Figure 5. The general steps are:1. Receive the alarm and notification from the monitored system, e.g., Live System Under Surveillance - LSUTS 401.2. Retrieve the time stamp of the event from the alarm and notification.3. Retrieve the data from raw dataset sources using the time stamp of the event and the slicing window policy.4. Apply the “Smart Slicing Time Window Mechanism” to generate the labelled data points according to the slicing window policy, referring to Figure 2.5. Store the labelled data into the storage.P112430W00113

[0077] The business logics in KBDSM module 406 is presented in Figure 6. The general steps are:1. Load the slicing window policy.2. Access the datasets by following the link to DSC described in the slicing window policy.3. Fetch all the qualified data points from the dataset by applying a “Target time period”.4. Sort out the events by scanning all the qualified data points using the event definition from DSC (e.g. key words, or description). This might be done through another LLM based model.5. Loop over all the identified events.6. Apply the “Smart Slicing Time Window Mechanism” to generate the labelled data points according to the slicing window policy, referring to Figure 2. 7. Verify if the number of the required data samples around the event is met or not.a. If no, go back to step 6.b. If yes, break and go to next step.8. Verify if this is the last event.a. If no, go back to step 5.b. If yes, go to next step.

[0078] Anomaly detection using the trained model is described next.

[0079] Turning to Figure 7, the proposed solution can be used in the network for anomaly detection. Any anomaly indication from the model inference is used to narrow down the evidence (e.g., time stamp) of when a suspected event occurred.

[0080] Two main components are used to do the anomaly detection in Figure 7. The domain specific trained model (DS-LLM-IM) 701, is deployed inside the anomaly detection for time series data module (AD4TSDM) 702 to provide the model inference. When an anomaly event is found, AD4TSDM 702 sends the notification / message to the special event trigger module (SETM) 703 to store the event information into the event-data storage (DS), called event database (DB) in figure 7, and triggers a notification towards the network engineer.

[0081] Figure 8 illustrates how to detect an anomaly event from the collected dataset. The detailed steps include:1. Fetch the data from dataset sources, e.g. log files.P112430W001142. Prepare the input data according to the slicing time window policy for model inference; and put the data frame into a data queue.3. Apply DS-LLM-IM to the data frames in the data queue.4. Verify all the outcomes to see if there is any outlier / anomaly.a. If yes, go to step 5.b. If no, go back to step 1 to fetch next trunk of data.5. Check if the time window size is under the range for the special event (time range), which is given in the event attribute under slicing time window for model inference.a. If no, go to step 6.b. If yes, all the special or anomaly events are found, go to step 7.6. Split the frame(s) associated to “anomaly outcome” and put them in the data queue, then go to step 3.7. The special or anomaly event(s) are located, and notification are sent to SETM (Special Event Trigger Module). The module continues to fetch next trunk of the data to handle (repeat step 1-7).

[0082] Experimental Results are discussed next.

[0083] A proof of concept (PoC) has been implemented and tested to demonstrate how the proposed data sampling mechanism works and what kind of outcome can be expected. The performance of the adapted model was compared to the manual operation done by the experience network engineers.

[0084] Implementation will now be discussed.

[0085] Referring to figure 9, a LLM (Llama 3 with 7B parameters) model 910 was fine-tuned and trained on distributed GPUs server 915. A large language model has enough capacity to learn complex patterns from massive dataset, which supports the present use case. Llama 3 was chosen as the base model because it is open-source and can be fine-tuned and re-trained offline. Therefore, a fine-tune Llama 3 was implemented for learning msg sequences on 4X VI 00 GPUs. With a fine-tuned Llama 3 model 920, the system had enough capability to learn massive log sequences.

[0086] With the novel dynamic slicing windows proposed in Figure 2, smart sampling policy, the time density statistics where one fail happens were precomputed, e.g., from certain log files, if the time interval density is condensed, then the dynamic windows size will be smaller, or vice versa. It was observed that in this way, the main root cause of a failure was better captured in time series.P112430W00115

[0087] The proposed solution applies a new way of sampling data to label failure messages from massive log lines based on dynamic slicing windows. The method can provide huge amount of training samples for large language models e.g., Llama 3 to learn the root cause of failures in complex radio log files.

[0088] The sampling / grouping techniques is explained in detail in Figure 10.

[0089] As the failure comes from previous behavior of log files as timeseries, it is assumed that a failure event occurs with some root cause that happens before the failure within a certain time window 1001, and the effect of the failed event will still last for some time after, as shown in the left-hand side of the figure 10. Two mechanisms have been designed using the slicing window to label failure / not failure events. One is with fixed time window which happens around a failure event, the other one is with dynamic time windows which can be learned by the model and statistics from each log file. More specifically, the window size that covers a failed event can vary from different log files. Therefore, a sequential log message vector can be formulated by this window. The right-hand side of the figure shows how grouped vectors have been designed and implemented, which can be injected into an LLM. Since a certain time period that happens before and after a failure event is leveraged, by the definition above, a fixed or a dynamic time window is applied. For the fixed time window, as an example, if the failure happens at time t, then N messages are included before t and M messages after t together as the total message which affects on the failure events. In this configuration, all N + M messages can be converted and concatenated as a single long vector V = [[tr], [v2]> ... [vw+M]] where each=vii>vi2> —vik\ where i G [1, N + M] as a fc-dimensional vector which is converted by the model from the raw message. Typically, k represents the dimension for the raw message in a high dimensional space. Therefore, the concatenated vector V will be a vector with (A + M)k dimensions. The concatenated vector V will be used as the input into an LLM.

[0090] The dataset and test environment are discussed next.

[0091] Referring to figure 11, it is important to understand data at business level and implement the logic for Al system to learn correctly. To this end, properly preparing and tailoring data from unstructured raw datasets for labeling data and input format becomes important. In the present PoC, each log file is organized as .json format. The ‘msg’ 1101 shows a message or an event that happens in that timestamp fromP112430W00116different processes. Each log file 1100 may contain millions of messages from different processes in time series order. The current Al system aims to detect or indicate a failure is about to occurs. For example, as is shown in figure 11, a failure / error event shows from process “AntModVswrCtrl”.

[0092] Referring to Figure 12, some representative examples of concatenated msg to indicate a failure / or not are provided. It demonstrates two scenarios:1. Before the failure occurs, the system msg shows multiple ‘forcing state started” injection into multiple branches. This msg chain is used as the input vector to label a ‘failure’;2. a regular msg chain that indicates the system is running normally.

[0093] Result analysis is discussed next.

[0094] Herein, the focus is directed towards the general solution architecture, from Figure 4, the log file data sampling, and the LLM training and inference pipeline. A distributed LLM was trained on a cluster with 4X VI 00 GPUs using the sampling and preprocessing method introduced previously. With dynamic slicing window, a huge number of labels were sampled for regular normal messages. For failed messages, using the predefined policy from business domain, the failure messages were sampled as much as possible. The reason a large language model such as Llama 3 was chosen is that pre-trained weights could be used, with >7B parameters, to handle a large number of sequences generated from log files.

[0095] Two Llama 3 models were trained using 400K input samples with 2 epochs on 4X GPUs on cluster. To avoid bias, 3 different test sets were made from training samples and the performance on each of them was evaluated.

[0096] Figure 13 shows the results with Fl score table. The First test set uses 20355 normal messages and 244 failure events, the model captured 236 out of 244 failures with much less false positive and negative. The second test set captured 34 failures out of 35 events, and the third test set captured 145 out of 148 failures. The results clearly show the fine-tuned LLM has enough capability to categorize failed / non-failed events with high Fl score in such an imbalanced dataset.

[0097] The results show reliable and significant robustness of detecting anomaly events with high Fl score and low false position / negative.

[0098] Potential business use cases are described next.

[0099] One use case for the solution provided herein is radio / baseband log file analysis. Radio products produce many different types of logs such as statis logs,P112430W00117proactive logs, timeseries logs which generates more than 50000 messages per log daily, it is a critical resource fortroubleshooting and debugging used by designers and testers. However, none of the existing solutions provide a unified and general solutions for processing these logs in a way that can be used by LLM. The solution proposed herein is generalized so that the framework and workflow can be applied on these business scenarios with very small adjustments.

[0100] Another use case can be the auto-labeling of failed events from massive proactive logs. Most of existing methods on log file analysis rely on the simple failure detection at single message level, and these methods are not fully implemented and functional. Whereas the solution described herein applies LLM to learn the failure event window which provide the pre-and post-root cause based on a failed event. The proposed solution provides to the end users not only where a failure event happens within a long log file, but also the time window that covers the cause of the failure. From the log file, it can be detected that some initializations occur before a failure happens and some postprocessing takes places after a short period of the failure. For example, after a channel stops unexceptionally, the system sends some alarm messages or postprocess notification on the log, that information is essential for an end-user to debug the failure in the future. The proposed solution can properly learn this critical information within a time window and feedback to the end user automatically.

[0101] Yet another business use case is a virtual operator, which helps an operator monitor the system’s health, predict outages by early diagnosis, and take actions accordingly. Currently, operators are monitoring live systems by looking at counters, alarms, stats, etc. All this does not prevent crashes / failures and outages because events are coming fast, and no current program is compiling all data real time to diagnose that a problem is coming. Currently no analysis in real time can detect if all is normal or if something bad is happening. The proposed solution provides a unified way of processing these massive time series data before feeding into advanced AI / ML models. Therefore, the solution described herein can be part of the monitoring system which collect counters, alarms, and logs together to associate them by learning failure event windows and provide or detect failure events.

[0102] Using large language model (LLM) has the advantage of taking massive message from all log files to train with one single model with enough capacity to predict events. A lot of data can be fed into the LLM because LLM hasP112430W00118billions of parameters to learn massive patterns. In general, the proposed solution with LLM as a base model is a generalized and powerful tool which can be applied to different businesses with a standard workflow and framework with minimal adjustment from different inputs.

[0103] The solution can be implemented in the cloud as a central service. It can be implemented as a radio access network (RAN) intelligent controller application (rApp) (Non- Real Time applications) in service management orchestration (SMO).

[0104] Turning to figure 14, there is provided a computer implemented method 1400 for training a large language model (LLM) to detect events in a time series. The method, comprises locating, step 1401, an event in a time series training dataset using a knowledge corpus. The method, comprises generating, step 1402, from the time series, time window samples having varying lengths and starting points, the time window samples including the event. The method, comprises extracting, step 1403, time series data, associated with the event, within the time window samples. The method, comprises training, step 1404, the LLM to detect events in the time series, using the event and the extracted time series data.

[0105] A knowledge corpus is a collection or body of knowledge or evidence. The knowledge corpus can comprise knowledge, in different formats, about events and related logs and / or activities. The knowledge corpus can be obtained, for example, from entry of data in a database, by engineers working in the present field. Alternatively, the knowledge corpus could be embodied as an Al agent performing a network engineer’s tasks, and / or developing products or solutions.

[0106] The knowledge corpus may be a database of events and associated time series data.

[0107] Locating an event may comprise extracting, step 1405, time series data associated with the event from the database and locating, step 1406, an instance of the time series data in the times series.

[0108] The method may further comprise computing, step 1407, an event start, and an event end based on information available in the knowledge corpus about the event.

[0109] Each time window sample may have a length and start defined based on a policy associated with the event or type of event.P112430W00119

[0110] Each time window sample may have a random length and random start, following any given distribution for sampling, and may be comprised within the period defined by the event start and the event end.

[0111] The method may further comprise computing, step 1408, a second event start, and a second event end based on information available in the knowledge corpus about the event.

[0112] The period defined by the second event start and the second event end may correspond to a precursor of the event and further time window samples may be generated and include the precursor of the event.

[0113] The further time window samples may have a random length and random start and may be comprised within the period defined by the event start and the event end.

[0114] The time series data associated with the event may comprise any one or more of: an activity type, an alarm, a notification, an identifier, and an error code.

[0115] The events may comprise any one or more of: central processing unit (CPU) overload, memory leak, overheat, packet loss, session establishment failure and data retransmission.

[0116] The time window samples may capture the trends or accumulation of the activities before the event or capture the trends or accumulation of the activities after the event.

[0117] The method may further comprise training, step 1409, the LLM to determine the lengths and starting points of the time window samples.

[0118] The method may further comprise using, step 1410, the trained LLM to detect events in the time series.

[0119] The method may further comprise using, step 1411, the trained LLM to forecast future events in the time series based on precursors of the future events.

[0120] It should be noted that methods and steps described herein are, generally, computer implemented methods and steps. The term computer may be interpreted as having different meanings, such as explained next, for example.

[0121] Referring to figure 15, there is provided an apparatus (HW) 1501, in which functions and steps described herein can be implemented.

[0122] The apparatus 1501 may be a server, a network node, a radio base station, or any other computing device which may be part of a cloud computing system, edge computing system, or which may be a standalone device. The apparatusP112430W001201501 is used to train an LLM and use the LLM to detect events and / or anomalies in times series. The technical effect is provided by automating this process and obtaining a large reduction of time to detect such events or anomalies. Further, systems using the trained LLM provided here can forecast future events or anomalies.

[0123] The apparatus 1501 comprises processing circuitry 1503 and memory 1505. The memory 1505 can contain instructions executable by the processing circuitry 1503 whereby functions and steps described herein may be executed to provide any of the relevant features and benefits disclosed herein.

[0124] The apparatus 1501 may also include non-transitory, persistent, machine-readable storage media 1507 having stored therein software and / or instruction 1509 executable by the processing circuitry 1503 to execute functions and steps described herein. The apparatus may also include network interface(s) and a power source.

[0125] The instructions 1509 may include a computer program for configuring the processing circuitry 1503. The computer program may be stored in a physical memory local to the device, which can be removable, or it could alternatively, or in part, be stored in the cloud. The computer program may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0126] Referring to figure 16, there is provided a virtualization environment 1600 in which functions and steps described herein can be implemented.

[0127] The virtualization environment 1600 (which may go beyond what is illustrated in figure 16), may comprise systems, networks, servers, nodes, devices, etc., that are in communication with each other either through wire or wirelessly, e.g. through a network interface component (NIC) comprising physical network interface(s). Some or all of the functions and steps described herein may be implemented as one or more virtual components (e.g., via one or more applications, components, functions, virtual machines, containers, etc.) executing on one or more physical apparatus in one or more networks, systems, environment, etc.

[0128] A virtualization environment provides hardware 1601 comprising processing circuitry 1603 and memory 1605. The memory 1605 can contain instructions executable by the processing circuitry 1603 whereby functions and steps described herein may be executed to provide any of the relevant features and benefits disclosed herein.P112430W00121

[0129] The hardware 1601 may also include non-transitory, persistent, machine-readable storage media 1607 having stored therein software and / or instruction 1609 executable by the processing circuitry 1603 to execute functions and steps described herein.

[0130] The instructions 1609 may include a computer program for configuring the processing circuitry 1603. The computer program may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program may be stored in a physical memory local to the hardware 1601, which can be removable, or it could alternatively, or in part, be stored in the cloud. The computer program may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0131] Referring again to figures 15 and 16, there is provided an apparatus 1501, 1601 operative to train a large language model (LLM) to detect events in a time series. The apparatus 1501, 1601 comprises processing circuits 1503, 1603 and a memory 1505, 1605, the memory containing instructions executable by the processing circuits whereby the apparatus is operative to locate an event in a time series training dataset using a knowledge corpus. The apparatus is operative to generate, from the time series, time window samples having varying lengths and starting points, the time window samples including the event. The apparatus is operative to extract time series data, associated with the event, within the time window samples. The apparatus is operative to train the LLM to detect events in the time series, using the event and the extracted time series data.

[0132] The apparatus is further operative to execute any of the steps described herein, for example it can execute all the steps and sub steps described in relation with the method of Figure 14.Referring again to figures 15 and 16, there is provided a non-transitory computer readable media 1507, 1607 having stored thereon instructions 1509, 1609 for training a large language model (LLM) to detect events in a time series. The instructions comprise locating an event in a time series training dataset using a knowledge corpus. The instructions comprise generating, from the time series, time window samples having varying lengths and starting points, the time window samples including the event. The instructions comprise extracting time series data, associated with the event, within the time window samples. The instructions compriseP112430W00122training the LLM to detect events in the time series, using the event and the extracted time series data.

[0133] The instructions may comprise further instructions for executing any of the steps described herein.

[0134] Modifications will come to mind to one skilled in the art having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that modifications, such as specific forms other than those described above, are intended to be included within the scope of this disclosure. The previous description is merely illustrative and should not be considered restrictive in any way. The scope sought is given by the appended claims, rather than the preceding description, and all variations and equivalents that fall within the range of the claims are intended to be embraced therein. Although specific terms may be employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

P112430W00123CLAIMS1. A computer implemented method for training a large language model (LLM) to detect events in a time series, comprising:locating an event in a time series training dataset using a knowledge corpus; generating, from the time series, time window samples having varying lengths and starting points, the time window samples including the event;extracting time series data, associated with the event, within the time window samples; andtraining the LLM to detect events in the time series, using the event and the extracted time series data.

2. The method of claim 1, wherein the knowledge corpus is a database of events and associated time series data.

3. The method of claim 2, wherein locating an event comprises:extracting time series data associated with the event from the database; and locating an instance of the time series data in the times series.

4. The method of claim 1, further comprising computing an event start, and an event end based on information available in the knowledge corpus about the event.

5. The method of claim 4, wherein each time window sample has a length and start defined based on a policy associated with the event or type of event.

6. The method of claim 4, wherein each time window sample has a random length and random start and is comprised within the period defined by the event start and the event end.

7. The method of claim 4, 5 or 6, further comprising computing a second event start, and a second event end based on information available in the knowledge corpus about the event.

8. The method of claim 7, wherein the period defined by the second event start and the second event end corresponds to a precursor of the event and wherein further time window samples are generated and include the precursor of the event.

9. The method of claim 7, wherein the further time window samples have a random length and random start and are comprised within the period defined by the event start and the event end.P112430W0012410. The method of any one of claims 1-9, wherein the time series data associated with the event comprises any one or more of: an activity type, an alarm, a notification, an identifier, and an error code.

11. The method of any one of claims 1-10, wherein the events comprise any one or more of: central processing unit (CPU) overload, memory leak, overheat, packet loss, session establishment failure and data retransmission.

12. The method of any one of claims 1-11, wherein the time window samples capture the trends or accumulation of the activities before the event or capture the trends or accumulation of the activities after the event.

13. The method of any one of claims 1-12, further comprising training the LLM to determine the lengths and starting points of the time window samples.

14. The method of any one of claims 1-13, further comprising using the trained LLM to detect events in the time series.

15. The method of any one of claims 1-14, further comprising using the trained LLM to forecast future events in the time series based on precursors of the future events.

16. An apparatus operative to train a large language model (LLM) to detect events in a time series comprising processing circuits and a memory, the memory containing instructions executable by the processing circuits whereby the apparatus is operative to:- locate an event in a time series training dataset using a knowledge corpus; - generate, from the time series, time window samples having varying lengths and starting points, the time window samples including the event;extract time series data, associated with the event, within the time window samples; and- train the LLM to detect events in the time series, using the event and the extracted time series data.

17. The apparatus of claim 16, further operative to execute any of the steps of claims 2 to 15.

18. A non-transitory computer readable media having stored thereon instructions for training a large language model (LLM) to detect events in a time series, the instructions comprising:- locating an event in a time series training dataset using a knowledge corpus;P112430W00125- generating, from the time series, time window samples having varying lengths and starting points, the time window samples including the event; extracting time series data, associated with the event, within the time window samples; and- training the LLM to detect events in the time series, using the event and the extracted time series data.

19. The non-transitory computer readable media of claim 18 further comprising instructions according to any one of claims 2 to 15.

Images