Method and apparatus for training anomalous traffic detection model, monitoring method, and device

Abstract

The present application relates to the technical field of traffic detection, and discloses a method and apparatus for training an anomalous traffic detection model, a monitoring method, and a device. The method comprises: acquiring a network traffic dataset, the network traffic dataset comprising a plurality of network traffic features; performing feature redundancy elimination on the network traffic dataset on the basis of the importance of each network traffic feature to form an alternative feature dataset; performing balancing processing on the alternative feature dataset to form a target feature dataset; and using the target feature dataset to train a preset network model to obtain an anomalous traffic detection model. In this way, screening network traffic features by means of the importance can retain only important network traffic features, so that the anomalous traffic detection model trained by means of the selected network traffic features is more targeted. Additionally, balancing processing is performed, so that the detection recall rate of small-sample network traffic feature types can be improved without increasing additional model complexity.

Description

Training methods and devices, monitoring methods and equipment for abnormal flow detection models

[0001] Cross-reference of related applications

[0002] This application claims priority to Chinese Patent Application No. 202411534575.1, filed on October 31, 2024, entitled “Training Method and Apparatus, Monitoring Method and Equipment for Abnormal Flow Detection Model”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of traffic detection technology, and in particular to a training method and apparatus, monitoring method and equipment for an abnormal traffic detection model. Background Technology

[0004] Network traffic, as the carrier of information transmission and interaction in cyberspace, contains a large amount of important information. With the continuous development of science and technology, network viruses and attack methods are becoming increasingly complex, resulting in some abnormal network traffic. Abnormal network traffic may threaten network security, so how to automatically and accurately identify abnormal network traffic data is an urgent problem to be solved.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This application provides a training method and apparatus for an abnormal traffic detection model, as well as a monitoring method and device, to automatically and accurately identify abnormal network traffic.

[0008] This application provides a training method for an abnormal traffic detection model, comprising: acquiring a network traffic dataset; the network traffic dataset including multiple network traffic features; performing feature redundancy elimination on the network traffic dataset according to the importance of each network traffic feature to form a candidate feature dataset; performing balancing processing on the candidate feature dataset to form a target feature dataset; and using the target feature dataset to train a preset network model to obtain an abnormal traffic detection model.

[0009] In the above embodiments, filtering network traffic features based on importance simplifies the network traffic features, retaining only the most important ones. This makes the anomaly traffic detection model trained using the filtered network traffic features more targeted. By balancing the candidate feature dataset to obtain the target feature dataset, the problem of data type imbalance in the candidate feature dataset can be reduced. This allows the model trained using the optimized candidate feature dataset to improve the detection recall rate for small sample network traffic feature types without increasing additional model complexity. Consequently, the anomaly traffic detection model trained using this method can automatically and accurately identify abnormal network traffic. Furthermore, the anomaly traffic detection model trained using this method can handle large-scale datasets, has a fast processing speed in practical applications, and can respond in real time.

[0010] Furthermore, acquiring a network traffic dataset includes: acquiring raw network data and data sources for each raw network data source; extracting key information from each raw network data source according to the data source; analyzing the correlation between each data source and fusing the correlation with each key information source to obtain several network traffic features; and the network traffic features constitute the network traffic dataset.

[0011] In the above implementation, by fusing the correlation and key information to obtain network traffic features, the information contained in the network traffic features can be made richer, so that the abnormal traffic detection model trained using the network traffic features can be more accurate.

[0012] Furthermore, based on the importance of each of the network traffic features, feature redundancy elimination is performed on the network traffic dataset to form a candidate feature dataset, including: step S11, inputting the network traffic dataset into a preset first limit tree model to obtain the importance score of each of the network traffic features; step S12, removing the network traffic feature with the lowest importance score from the network traffic dataset to obtain a new network traffic dataset; repeating steps S11 to S12 until a preset condition is met, and the remaining network traffic features form the candidate feature dataset; the preset condition includes: the number of network traffic features in the candidate feature dataset is less than or equal to a preset number.

[0013] In the above implementation, important network traffic features often have a greater impact on whether the predicted traffic is abnormal. By selecting important network traffic features to form a candidate feature dataset, the accuracy of the trained abnormal traffic detection model can be improved. At the same time, by selecting important network traffic features, the number of features used for training can be reduced, thereby reducing the resources and time required for model training.

[0014] Furthermore, the process of eliminating feature redundancy in the network traffic dataset and forming a candidate feature dataset based on the importance of each network traffic feature includes: generating shadow features with the same data distribution characteristics as the network traffic features; constructing several decision trees using a preset second limit tree model, calculating the Gini index reduction of each network traffic feature when splitting a node in each decision tree, and selecting the network traffic feature with the largest Gini index reduction for splitting, and calculating the importance score of each network traffic feature based on all Gini index reductions in the second limit tree model; constructing several decision trees using a preset third limit tree model, calculating the Gini index reduction of each shadow feature when splitting a node in each decision tree, and selecting the shadow feature with the largest Gini index reduction for splitting, and calculating the importance score of each shadow feature based on all Gini index reductions in the third limit tree model; and selecting a portion of the network traffic features based on the importance scores of the network traffic features and the shadow features to form the candidate feature dataset.

[0015] In the above implementation, the importance score of network traffic features reflects their actual role in the model, while the importance score of shadow features should be close to zero because shadow features do not contain any actual information. Therefore, by comparing the importance scores of network traffic features and shadow features, the salience of each network traffic feature can be evaluated, which in turn helps to make the subsequently trained abnormal traffic detection model more accurate.

[0016] Furthermore, filtering some of the network traffic features and forming the candidate feature dataset based on the importance scores of the network traffic features and the shadow features includes: obtaining the maximum value among the importance scores of all the shadow features; comparing the importance score of each network traffic feature with the maximum value, and retaining the network traffic features with importance scores greater than the maximum value to form the candidate feature dataset.

[0017] In the above implementation, network traffic features that are significantly more important than shadow features are considered global features that significantly contribute to model prediction. Therefore, network traffic features with importance scores greater than the maximum value of shadow features are retained and used to form a candidate feature dataset. This ensures that the network traffic features used in subsequent model training all significantly contribute to model prediction, thereby improving the accuracy of the trained abnormal traffic detection model.

[0018] Furthermore, the candidate feature dataset is balanced to form a target feature dataset, including: for each network traffic feature: determining the distance between the network traffic feature and other network traffic features; identifying network traffic features with a distance less than a preset threshold as nearest neighbor samples; counting the types of each nearest neighbor sample and the number of each type, defining the type with the most numbers as the dominant type; if the dominant type is consistent with the type of the network traffic feature, then retaining the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, then deleting the network traffic feature or adjusting the preset weight of the network traffic feature, and the remaining network traffic features form the target feature dataset.

[0019] In the above implementation, if the dominant type and the network traffic feature type are different, the network traffic feature may be an edge point or a noise point, which will cause data imbalance. Therefore, by determining the dominant type corresponding to the network traffic feature and deleting network traffic features that are inconsistent with the dominant type, the problem of data imbalance in the candidate feature dataset can be reduced. This allows the model to be trained using the optimized candidate feature dataset, thereby improving the detection recall rate of small sample network traffic feature types without increasing the additional model complexity.

[0020] Furthermore, the preset network model is a multilayer perceptron model; training the preset network model using the target feature dataset to obtain an abnormal traffic detection model includes: inputting the target feature dataset into the multilayer perceptron model for training, and using a multi-head attention mechanism to perform weighted summation of the features in the target feature dataset to obtain an output result; calculating the loss value of the output result according to a preset loss function; optimizing the weights and biases of each layer in the multilayer perceptron model according to the loss value and a preset optimization algorithm until convergence or reaching a set number of iterations to obtain the abnormal traffic detection model.

[0021] In the above implementation, by utilizing a multi-head attention mechanism to perform weighted summation of features in the target feature dataset, the importance of different features can be dynamically adjusted using attention weights. This enables adaptive focusing on the network traffic features most critical for detecting abnormal traffic, resulting in a more accurate abnormal traffic detection model.

[0022] This application provides an abnormal traffic monitoring method, including: acquiring traffic to be detected; inputting the traffic to be detected into a preset abnormal traffic detection model to determine whether the traffic to be detected is abnormal; the abnormal traffic detection model is obtained through the above-described abnormal traffic detection model training method.

[0023] In the above implementation, abnormal traffic detection model can effectively handle large-scale datasets, identify abnormal traffic quickly, and respond in real time.

[0024] This application provides a training device for an abnormal traffic detection model, comprising: a dataset acquisition module for acquiring a network traffic dataset; the network traffic dataset includes multiple network traffic features; a first optimization module for performing feature redundancy elimination on the network traffic dataset according to the importance of each network traffic feature and forming a candidate feature dataset; a second optimization module for performing balancing processing on the candidate feature dataset and forming a target feature dataset; and a training module for training a preset network model using the target feature dataset to obtain an abnormal traffic detection model.

[0025] This application provides an electronic device, including a processor and a memory. The memory stores computer-executable instructions that can be executed by the processor. The processor executes the computer-executable instructions to implement the above-described abnormal traffic detection model training method or the above-described abnormal traffic monitoring method.

[0026] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0027] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are considered similar elements. The drawings do not constitute a limitation of scale, and wherein:

[0028] Figure 1 is a schematic diagram of a training method for an abnormal traffic detection model provided in an embodiment of this application;

[0029] Figure 2 is a schematic diagram of an abnormal traffic monitoring method provided in an embodiment of this application;

[0030] Figure 3 is a schematic diagram of a training device for an abnormal traffic detection model provided in an embodiment of this application;

[0031] Figure 4 is a schematic diagram of an abnormal flow monitoring device provided in an embodiment of this application;

[0032] Figure 5 is a schematic diagram of an electronic device provided in an embodiment of this application.

[0033] Icons: Dataset acquisition module 1; First optimization module 2; Second optimization module 3; Training module 4; Traffic acquisition module 5; Anomaly determination module 6; Memory 7; Communication interface 8; Bus 9; Processor 10. Detailed Implementation

[0034] To provide a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0035] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0036] Unless otherwise stated, the term "multiple" means two or more.

[0037] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0038] Example 1

[0039] This application provides a training method for an abnormal traffic detection model. Referring to Figure 1, which is a basic flowchart of the training method for the abnormal traffic detection model provided in this application, it includes:

[0040] Step S101: Obtain network traffic dataset.

[0041] The network traffic dataset includes multiple network traffic features.

[0042] In some embodiments, step S101 may include: acquiring raw network data and the data sources of each raw network data; extracting key information from each raw network data according to the data sources; analyzing the correlation between each data source and integrating the correlation with each key information to obtain several network traffic features; and the network traffic features constitute a network traffic dataset.

[0043] In the above embodiments, the raw network data can be network traffic data. For example, network traffic data includes at least one of the following: IP address (Internet Protocol address), port number, protocol type, packet content, session duration, packet size distribution, log information from different network layers, security device alarms, and status information of other network security assets. The network layers include, for example, the application layer, transport layer, and network layer.

[0044] In one optional embodiment of the above, obtaining the original network data may be: obtaining preprocessed original network data.

[0045] Among the above-mentioned optional methods, the raw network data can be preprocessed in the following ways:

[0046] Method 1: Standardize the format or units of the raw data from each network.

[0047] It is understandable that data from different network devices or at different times may have different formats or units. The format or unit of the raw data from each network can be identified, and through data conversion or standardization operations, the format or unit of the raw data from each network can be unified to ensure the consistency of the raw data from each network in the dataset. For example, if the raw network data has different timestamp formats, the raw network data can be converted into a unified time format. This unified time format can be a pre-defined time format.

[0048] Method 2: Obtain the data type of the original network data, search for the corresponding conversion type in the preset type database, and convert the original network data according to the conversion type.

[0049] For example, text logs are converted into structured data tables. Hexadecimal values ​​are converted into decimal values. Non-numeric data, such as protocol types and service types, is encoded into numeric data. Performing necessary data type conversions on captured raw network data facilitates better processing and analysis by machine learning algorithms.

[0050] Method 3: Determine if there are outliers in the original network data, and correct the original network data with outliers.

[0051] For example, statistical analysis and data mining techniques can be applied to identify and handle outliers. For instance, box plots, Z-scores, or IQR (interquartile range) methods can be used to process raw network data to identify outliers that significantly deviate from the normal range. These outliers can then be corrected, deleted, or replaced to prevent them from adversely affecting subsequent analysis. Outlier detection algorithms can also be used to handle outliers to improve the accuracy of subsequent analysis.

[0052] Method 4: When the original network data is time series data, sort the original network data in chronological order, and ensure that the time intervals between the original network data are the same.

[0053] It is understandable that, since the time intervals of the original network data may differ, techniques such as interpolation, data resampling, or time window aggregation can be used to add or subtract from the original network data to ensure the continuity and comparability of the time series data.

[0054] In one optional embodiment of the above, extracting key information from each network's raw data according to the data source can be done by: selecting available data sources as alternative data sources from the data sources; and extracting key information from the network's raw data corresponding to the alternative data sources.

[0055] Among the above optional methods, selecting available data sources as alternative data sources can be done by comparing the data source with a preset data source, and confirming the data source as an alternative data source if the data source matches the preset data source.

[0056] The default data sources can be set by engineers based on their experience. Examples of default data sources include: network traffic logs, Security Event Management System (SIEM) records, firewall and Intrusion Detection System (IDS / IPS) alarms, application logs, system logs, and user behavior data.

[0057] In the above optional methods, extracting key information from the original network data corresponding to the alternative data source can be done by inputting the original network data corresponding to the alternative data source into a preset key information extraction model to obtain the key information from the original network data.

[0058] One way to obtain a key information extraction model is to acquire raw sample network data with key information labels, input the raw sample network data with key information labels into a pre-set training model for training, and then obtain the key information extraction model.

[0059] Among the above optional methods, specific parsing techniques can also be applied to alternative data sources, such as text parsing and regular expressions, event correlation and behavior analysis, data parsing and natural language processing, to extract key information.

[0060] Critical information refers to core data or content extracted from raw network data that is essential for understanding and analyzing network traffic characteristics. Examples of critical information include: structured data, such as tabular data from a data source; unstructured data, such as important text content in log files; traffic statistics, such as statistical information extracted from network traffic logs; security incident information, such as relevant information extracted from Security Incident Management System (SIEM) records; user behavior data, such as access patterns extracted from user behavior data; device alarm information, such as alarms extracted from firewalls and Intrusion Detection Systems (IDS / IPS); and critical information in application and system logs.

[0061] In one optional embodiment of the above, analyzing the relationships between data sources can be done by sending each data source to a preset analysis platform and obtaining the relationships between the data sources as fed back by the analysis platform. The analysis platform is used to annotate the relationships between the data sources in response to user actions.

[0062] In another optional embodiment described above, analyzing the relationship between data sources may involve: obtaining key information corresponding to each data source; constructing features that can characterize the attributes of each key information source; and using the features to perform calculations based on a preset statistical algorithm to obtain the relationship between the data sources.

[0063] This process of constructing features that characterize each key information attribute can include: performing time-series alignment on each key information item, and then constructing features for the time-series aligned key information item. In this way, time-series alignment of data from all data sources allows for the analysis of relationships between different data sources within the same time frame.

[0064] It's understandable that aligning key information across time series can involve standardizing all key information to the same time granularity, such as per minute or per hour. Furthermore, for missing time points, interpolation methods can be used to fill in the missing information or mark them as missing.

[0065] For example, a set of descriptive features can be created for each data source, representing the attributes of that data source. For instance, for network traffic data, these could include attributes such as total traffic volume, number of unique IPs, and protocol distribution; for security incident data, they could include attributes such as incident type distribution and severity distribution. It can be understood that the descriptive features can be represented as feature vectors, which contain various elements representing the attributes of the data source. Engineers can pre-define the attributes corresponding to each data source and construct the descriptive features according to these pre-defined attributes.

[0066] The process involves using a pre-defined statistical algorithm to calculate the correlation between data sources based on features. This can be achieved by: 1) Calculating the Pearson correlation coefficient between two features to determine the linear correlation between the two data sources corresponding to those features, when the features are numerical; 2) Using the chi-square test to assess the correlation between the data sources corresponding to the two features, when the features are categorical; and 3) Using mutual information to capture non-linear relationships between the data sources corresponding to the two features.

[0067] Among the above-mentioned alternative methods, time lag analysis can also be performed on each data source. This involves using cross-correlation functions to detect time dependencies between different data sources and to identify the time lag corresponding to the strongest correlation. In this way, since certain events may cause changes in other data sources, time lag analysis may reveal causal relationships between them.

[0068] For example, analyzing the correlation between various data sources, such as correlating network traffic data with security alerts, can help identify potential attack patterns or malicious activities.

[0069] In the above optional methods, after obtaining the correlation between various data sources, a graphical model can be constructed to represent the relationships between data sources. For example, by using data sources as nodes, using correlation strength as edge weights, and using time lag information to determine the direction of edges without conflict, a graphical model can be obtained. Community detection algorithms are applied to the constructed graphical model: algorithms such as the Louvain algorithm (a community detection algorithm) or label propagation algorithms are used to identify closely connected groups of data sources, which may represent sets of highly correlated data sources. Anomaly detection is performed on the constructed graphical model by identifying edges with abnormal correlation strengths, and / or by identifying connections that do not conform to the overall pattern. Here, abnormal correlation strength refers to a correlation higher than a first preset correlation or lower than a second preset correlation. The first preset correlation is greater than the second preset correlation.

[0070] Among the above alternatives, considering that the relationships between data sources may change over time, the sliding window method can also be used to repeat the above analysis, track changes in relationship strength and patterns, and identify relationship abrupt change points, which may indicate the occurrence of important events.

[0071] In addition to the above optional methods, a relationship summary can also be generated based on the relationships obtained. The relationship summary can include the strongest associations, key groups, and anomalies, and automatically classifies the identified relationships, such as "strong positive correlation," "weak negative correlation," and "periodic association." This allows for the integration of all analysis results, forming a comprehensive understanding of the relationships between data sources.

[0072] In the above embodiments, the correlation relationship is fused with each key information to obtain several network traffic characteristics. This can be achieved by using data fusion technology to integrate key information from different sources.

[0073] Network traffic features can be understood as a comprehensive representation that includes key information extracted from one or more related raw network data, as well as the relationships between the data sources to which this key information belongs and other data sources. Its representation can be a multidimensional matrix or tensor, where each dimension can represent different features or attributes. That is, network traffic features can be a comprehensive representation including a single key piece of information and the relationships between the data source to which that key information belongs and other data sources. Alternatively, network traffic features can be a comprehensive representation including multiple related key pieces of information and the relationships between the data sources to which each key piece of information belongs and other data sources.

[0074] For example, it can be assumed that there is a correlation between the original network data corresponding to linearly correlated data sources.

[0075] For example, network traffic characteristics can include information content, a time dimension, and correlation strength. Information content may include the core content of the original key information, as well as derived information obtained from correlation analysis. The time dimension may include time-series information reflecting the data state at different points in time. Correlation strength may include quantitative indicators representing the strength of correlations between different data sources. Data fusion techniques, such as data alignment, data fusion frameworks, and entity resolution, may be involved. The entire data fusion process may involve steps such as time synchronization, data format standardization, entity identification, and relationship construction.

[0076] In the above embodiments, after analyzing the relationships between various data sources and integrating these relationships with key information, the method further includes: constructing a comprehensive network traffic data view based on the integrated data. This constructed network traffic data view will contain rich contextual information, such as user identity, device type, application behavior, network protocol usage, and the relationships between them. This allows users to intuitively understand network traffic data.

[0077] For example, to improve the accuracy of subsequent feature extraction and analysis, the data view can be further enriched by introducing contextual information such as threat intelligence, historical security events, and user behavior benchmarks.

[0078] Step S102: Based on the importance of each network traffic feature, feature redundancy elimination is performed on the network traffic dataset to form a candidate feature dataset.

[0079] In some embodiments, step S102 may include: step S11, inputting the network traffic dataset into a preset first limit tree model to obtain the importance score of each network traffic feature; step S12, removing the network traffic feature with the lowest importance score from the network traffic dataset to obtain a new network traffic dataset; repeating steps S11 to S12 until a preset condition is met, and the remaining network traffic features form a candidate feature dataset; the preset condition includes: the number of network traffic features in the candidate feature dataset is less than or equal to a preset number.

[0080] In the above embodiments, inputting the network traffic dataset into the preset first limit tree model can be achieved by processing the network traffic dataset using a PCA (Principal Component Analysis) algorithm or an autoencoder algorithm, and then inputting the processed network traffic dataset into the preset first limit tree model. In this way, the PCA algorithm or autoencoder algorithm can reduce the dimensionality of the network traffic features in the network traffic dataset, mapping the high-dimensional feature space to a lower-dimensional space while retaining key information.

[0081] In the above embodiments, the preset condition can also be that the model performance meets a specific standard. The model performance can be at least one of accuracy, recall, and F1 score. For example, meeting a specific standard could mean that the accuracy reaches a set accuracy rate. Meeting a specific standard could also mean that the recall rate reaches a set recall rate. Meeting a specific standard could also mean that the F1 score reaches a set F1 score.

[0082] The network traffic dataset can be divided into training and testing sets. The first limit tree model can be tested using the testing set to obtain true positives, true negatives, false positives, and false negatives. Precision = (True Positives + True Negatives) / (True Positives + True Negatives + False Positives + False Negatives). Recall = True Positives / (True Positives + False Negatives). F1 Score = 2 * (Precision * Recall) / (Precision + Recall). True positives are the number of samples correctly predicted as positive. True negatives are the number of samples correctly predicted as negative. False positives are the number of samples incorrectly predicted as positive. False negatives are the number of samples incorrectly predicted as negative. "*" represents multiplication.

[0083] In some embodiments, step S102 may further include: generating shadow features with the same data distribution characteristics as network traffic features; constructing several decision trees using a preset second limit tree model, calculating the Gini index reduction of each network traffic feature when splitting a node in each decision tree, and selecting the network traffic feature with the largest Gini index reduction for splitting, and calculating the importance score of each network traffic feature based on all Gini index reductions in the second limit tree model; constructing several decision trees using a preset third limit tree model, calculating the Gini index reduction of each shadow feature when splitting a node in each decision tree, and selecting the shadow feature with the largest Gini index reduction for splitting, and calculating the importance score of each shadow feature based on all Gini index reductions in the third limit tree model; and selecting a subset of network traffic features based on the importance scores of the network traffic features and shadow features to form a candidate feature dataset.

[0084] Data distribution characteristics can refer to the mean, median, mode, skewness coefficient, etc. For example, a shadow feature with the same data distribution characteristics as network traffic characteristics can be generated using a random measurement algorithm.

[0085] For example, the Gini index can be calculated in the following way: Where, p i is the proportion of samples in class i in the dataset, and m is the total number of classes. The decrease in the Gini index can be calculated as follows:

[0086] Where D is the dataset of the current node, D j It is the subset obtained after splitting, |D j |Is D j The number of samples, |D| is the total number of samples in the current node's dataset.

[0087] It's understandable that for a dataset containing mixed classes, a higher Gini index indicates lower dataset purity; conversely, a lower Gini index indicates the highest dataset purity when all samples belong to the same class. The reduction in Gini index is a crucial metric in feature selection and decision tree construction. It helps evaluate the effectiveness of splitting and guides the growth direction of the decision tree to build an efficient classification model.

[0088] In the above embodiments, filtering network traffic features and forming a candidate feature dataset based on the importance scores of network traffic features and shadow features may include: obtaining the maximum value among all shadow feature importance scores; comparing the importance score of each network traffic feature with the maximum value, and retaining network traffic features with importance scores greater than the maximum value to form the candidate feature dataset. Since shadow features do not contain any actual information, their importance should be close to zero. Therefore, network traffic features with importance scores greater than the maximum value among shadow feature importance scores are more important than other network traffic features, thus enabling the selection of more important network traffic features.

[0089] Step S103: The candidate feature dataset is balanced to form the target feature dataset.

[0090] In some embodiments, step S103 may include: for each network traffic feature: determining the distance between the network traffic feature and other network traffic features; identifying network traffic features with a distance less than a preset threshold as nearest neighbor samples; counting the types of each nearest neighbor sample and the number of each type, defining the type with the most number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, then retaining the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, then deleting the network traffic feature or adjusting the preset weight of the network traffic feature, and the remaining network traffic features form the target feature dataset.

[0091] In the above embodiments, the distance between two network traffic features can be determined by Euclidean distance, Chebyshev distance, etc.

[0092] It is understandable that, before calculating the distance for each network traffic feature, one-hot coding can be used to re-encode each network traffic feature. The distance can then be calculated using the encoded network traffic features.

[0093] Step S104: Train the preset network model using the target feature dataset to obtain the abnormal traffic detection model.

[0094] In some embodiments, the preset network model is a multilayer perceptron model; step S104 may include: inputting the target feature dataset into the multilayer perceptron model for training, and using a multi-head attention mechanism to perform weighted summation of the features in the target feature dataset to obtain the output result; calculating the loss value of the output result according to a preset loss function; optimizing the weights and biases of each layer in the multilayer perceptron model according to the loss value and a preset optimization algorithm until convergence or reaching a set number of iterations to obtain an abnormal traffic detection model.

[0095] This includes loss functions, such as cross-entropy loss functions, and preset optimization algorithms, such as gradient descent and Adam algorithms (an optimization algorithm combining momentum and adaptive learning rates).

[0096] For example, a multilayer perceptron consists of multiple hidden layers, each containing several neurons. Each neuron receives input from the previous layer, performs a nonlinear transformation on the input data using an activation function, and then passes the result to the next layer. Simultaneously, a multi-head attention mechanism is introduced in each hidden layer, where each attention head independently weights the input features. This stacked approach enables the model to learn complex nonlinear relationships, improving its ability to map network traffic data.

[0097] For example, firstly, multiple data acquisition nodes are deployed at key nodes in the network to capture network traffic data in real time. These data acquisition nodes include, for example, routers, switches, firewalls, and intrusion detection systems. The collected network traffic data includes, but is not limited to, key traffic characteristics such as IP addresses, port numbers, protocol types, packet sizes, and flow durations. The collected network traffic data undergoes preprocessing, which may include removing duplicate records, filling missing values, data type conversion, normalization, and standardization. Specifically, min-max normalization and Z-score normalization methods can be used to ensure the consistency and comparability of the network traffic data. Outlier detection algorithms are applied to handle abnormal values ​​to improve the accuracy of subsequent analysis. Then, key information is extracted from each network traffic data source, and the key information and the correlation between data sources are used as network traffic features, forming a network traffic dataset. Finally, a network traffic feature redundancy elimination method based on limit tree recursive elimination (ET-RFE) is used to evaluate the importance of network traffic features using the limit tree algorithm. Candidate feature datasets are selected by counting the number of network traffic features in the network traffic dataset or by using cross-validation (CV). Then, a fully relevant salient feature selection method based on ET-Boruta (a feature selection method based on random forest) is applied to generate shadow features through a randomization strategy. The importance of these shadow features is compared with that of the network traffic features in the dataset to evaluate their significance. The more important network traffic features are retained as candidate feature datasets. Next, a data equalization method based on All-KNN (K-Nearest Neighbors) is implemented. This involves reconstructing the network traffic features using one-hot encoding. For each reconstructed feature, nearest neighbors are determined, and the type of these neighbors determines whether to retain the feature. The retained network traffic features form the target feature dataset. Finally, an abnormal traffic detection model based on a multi-head attention mechanism multilayer perceptron (MHA-MLP) is constructed. The target feature dataset is input into the first hidden layer of the MHA-MLP model. Multiple attention heads are applied to independently weight the features, and the weights and biases are optimized using backpropagation and gradient descent to obtain the final abnormal traffic detection model.

[0098] Example 2

[0099] This application provides an abnormal traffic monitoring method. Referring to Figure 2, which is a basic flowchart of the abnormal traffic monitoring method provided in this application, it includes:

[0100] Step S201: Obtain the traffic to be detected.

[0101] In some embodiments, the traffic to be detected is network traffic data.

[0102] For example, data acquisition nodes can be deployed at key nodes in the network to capture network traffic data in real time. These data acquisition nodes can be, for example, routers, switches, firewalls, and intrusion detection systems. The data acquisition nodes are used to collect network traffic data.

[0103] Step S202: Input the traffic to be detected into the preset abnormal traffic detection model to obtain whether the traffic to be detected is abnormal.

[0104] The abnormal traffic detection model was obtained using the method described in Example 1.

[0105] In some embodiments, inputting the traffic to be detected into a preset abnormal traffic detection model to determine whether the traffic to be detected is abnormal can be: preprocessing the traffic to be detected; inputting the preprocessed traffic to be detected into the preset abnormal traffic detection model to determine whether the traffic to be detected is abnormal.

[0106] The method for preprocessing the traffic to be detected can be found in the technical means of preprocessing the original network data in Example 1.

[0107] Example 3

[0108] Based on the same inventive concept, this application provides a training device for an abnormal traffic detection model, as shown in Figure 3. The training device for the abnormal traffic detection model includes: a dataset acquisition module 1, a first optimization module 2, a second optimization module 3, and a training module 4. The dataset acquisition module 1 is used to acquire a network traffic dataset; the network traffic dataset includes multiple network traffic features; the first optimization module 2 is used to eliminate feature redundancy in the network traffic dataset according to the importance of each network traffic feature and form a candidate feature dataset; the second optimization module 3 is used to perform balancing processing on the candidate feature dataset and form a target feature dataset; the training module 4 is used to train a preset network model using the target feature dataset to obtain the abnormal traffic detection model.

[0109] In some embodiments, the dataset acquisition module 1 is used to acquire network traffic datasets by: acquiring raw network data and the data sources of each raw network data; extracting key information from each raw network data according to the data sources; analyzing the correlation between each data source and integrating the correlation with each key information to obtain several network traffic features; and the network traffic features constitute a network traffic dataset.

[0110] In some embodiments, the first optimization module 2 is used to eliminate feature redundancy in the network traffic dataset and form a candidate feature dataset according to the importance of each network traffic feature in the following manner: Step S11, input the network traffic dataset into a preset first limit tree model to obtain the importance score of each network traffic feature; Step S12, remove the network traffic feature with the lowest importance score in the network traffic dataset to obtain a new network traffic dataset; repeat steps S11 to S12 until a preset condition is met, and the remaining network traffic features form a candidate feature dataset; the preset condition includes: the number of network traffic features in the candidate feature dataset is less than or equal to a preset number.

[0111] In some embodiments, the first optimization module 2 is used to eliminate feature redundancy in the network traffic dataset and form a candidate feature dataset based on the importance of each network traffic feature in the following manner: generating shadow features with the same data distribution characteristics as the network traffic features; constructing several decision trees using a preset second limit tree model, calculating the Gini index reduction of each network traffic feature when splitting the nodes of each decision tree, and selecting the network traffic feature with the largest Gini index reduction for splitting, and calculating the importance score of each network traffic feature based on all Gini index reductions in the second limit tree model; constructing several decision trees using a preset third limit tree model, calculating the Gini index reduction of each shadow feature when splitting the nodes of each decision tree, and selecting the shadow feature with the largest Gini index reduction for splitting, and calculating the importance score of each shadow feature based on all Gini index reductions in the third limit tree model; and selecting some network traffic features based on the importance scores of the network traffic features and shadow features to form a candidate feature dataset.

[0112] In some embodiments, the first optimization module 2 is used to filter some network traffic features and form a candidate feature dataset based on the importance scores of network traffic features and shadow features in the following manner: obtaining the maximum value among the importance scores of all shadow features; comparing the importance score of each network traffic feature with the maximum value, and retaining the network traffic features with an importance score greater than the maximum value to form a candidate feature dataset.

[0113] In some embodiments, the second optimization module 3 is used to balance the candidate feature dataset and form a target feature dataset in the following manner: For each network traffic feature: determine the distance between the network traffic feature and other network traffic features; determine network traffic features with a distance less than a preset threshold as nearest neighbor samples; count the types of each nearest neighbor sample and the number of each type, and define the type with the most number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, retain the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, delete the network traffic feature or adjust the preset weight of the network traffic feature, and the remaining network traffic features form the target feature dataset.

[0114] In some embodiments, the preset network model is a multilayer perceptron model; the training module 4 is used to train the preset network model using the target feature dataset in the following manner to obtain an abnormal traffic detection model: inputting the target feature dataset into the multilayer perceptron model for training, and using a multi-head attention mechanism to perform weighted summation of the features in the target feature dataset to obtain the output result; calculating the loss value of the output result according to the preset loss function; optimizing the weights and biases of each layer in the multilayer perceptron model according to the loss value and the preset optimization algorithm until convergence or reaching the set number of iterations to obtain an abnormal traffic detection model.

[0115] It is understood that the embodiments described in Embodiment 1 are also applicable to Embodiment 3 without conflict. For the sake of brevity, they will not be repeated here.

[0116] Example 4

[0117] Based on the same inventive concept, this application provides an abnormal traffic monitoring device, as shown in FIG4. The abnormal traffic monitoring device includes a traffic acquisition module 5 and an anomaly determination module 6. The traffic acquisition module 5 is used to acquire the traffic to be detected; the anomaly determination module 6 is used to input the traffic to be detected into a preset abnormal traffic detection model to determine whether the traffic to be detected is abnormal.

[0118] Example 5

[0119] Referring to Figure 5, this application embodiment provides an electronic device including a processor 10 and a memory 7. Optionally, the device may further include a communication interface 8 and a bus 9. The processor 10, communication interface 8, and memory 7 can communicate with each other via the bus 9. The communication interface 8 can be used for information transmission. The processor 10 can call logical instructions in the memory 7 to execute the training method or abnormal traffic monitoring method of the abnormal traffic detection model described in the above embodiment.

[0120] Furthermore, the logical instructions in the aforementioned memory 7 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0121] The memory 7, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 10 executes functional applications and data processing by running the program instructions / modules stored in the memory 7, that is, implementing the training method or abnormal traffic monitoring method of the abnormal traffic detection model in the above embodiments.

[0122] The memory 7 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 7 may include high-speed random access memory and may also include non-volatile memory.

[0123] Among them, electronic devices can be computers or servers, etc.

[0124] This application provides a storage medium storing computer-executable instructions, which are configured to execute the training method or the abnormal traffic detection method of the above-described abnormal traffic detection model.

[0125] This application provides a computer program product, which includes a computer program stored on a storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform the training method or abnormal traffic monitoring method of the above-mentioned abnormal traffic detection model.

[0126] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0127] The technical solutions of this application embodiment can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this application embodiment. The aforementioned storage medium can be a non-transitory storage medium, including various media capable of storing program code such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks, or it can be a transient storage medium.

[0128] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0129] The above descriptions are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. Furthermore, the above embodiments can be combined with each other to form new embodiments without conflict.

Claims

1. A training method for an abnormal traffic detection model, characterized in that, include: Obtain network traffic dataset; The network traffic dataset includes multiple network traffic features; Based on the importance of each network traffic feature, feature redundancy elimination is performed on the network traffic dataset to form a candidate feature dataset; The candidate feature datasets are balanced to form the target feature dataset; The pre-defined network model is trained using the target feature dataset to obtain an abnormal traffic detection model; The process of balancing the candidate feature datasets to form a target feature dataset includes: for each network traffic feature: determining the distance between the network traffic feature and other network traffic features; identifying network traffic features with a distance less than a preset threshold as nearest neighbor samples; counting the types of each nearest neighbor sample and the number of each type, defining the type with the most numbers as the dominant type; if the dominant type is consistent with the type of the network traffic feature, then retaining the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, then deleting the network traffic feature or adjusting the preset weight of the network traffic feature, and the remaining network traffic features forming the target feature dataset.

2. The method according to claim 1, characterized in that, Obtain network traffic datasets, including: Obtain the raw network data and the data source for each of the raw network data; Extract key information from each of the original network data sources according to the data sources; The relationships between the various data sources are analyzed, and these relationships are integrated with the key information to obtain several network traffic features; the network traffic features constitute the network traffic dataset.

3. The method according to claim 1, characterized in that, Based on the importance of each of the aforementioned network traffic features, feature redundancy is eliminated from the network traffic dataset to form a candidate feature dataset, including: Step S11: Input the network traffic dataset into a preset first limit tree model to obtain the importance score of each network traffic feature; Step S12: Remove the network traffic feature with the lowest importance score from the network traffic dataset to obtain a new network traffic dataset; Repeat steps S11 to S12 until a preset condition is met, and the remaining network traffic features form the candidate feature dataset; the preset condition includes: the number of network traffic features in the candidate feature dataset is less than or equal to a preset number.

4. The method according to claim 1 or 3, characterized in that, Based on the importance of each of the aforementioned network traffic features, feature redundancy elimination is performed on the network traffic dataset to form a candidate feature dataset, which also includes: Generate shadow features that have the same data distribution characteristics as the network traffic features; Several decision trees are constructed using a pre-defined second limit tree model. When splitting a node in each decision tree, the Gini index reduction of each network traffic feature is calculated, and the network traffic feature with the largest Gini index reduction is selected for splitting. The importance score of each network traffic feature is calculated based on all the Gini index reductions of each network traffic feature in the second limit tree model. Several decision trees are constructed using a pre-defined third limit tree model. When splitting a node in each decision tree, the reduction in Gini index of each shadow feature is calculated, and the shadow feature with the largest reduction in Gini index is selected for splitting. The importance score of each shadow feature is calculated based on all the reductions in Gini index of each shadow feature in the third limit tree model. Based on the importance scores of the network traffic features and the shadow features, a subset of the network traffic features are selected to form the candidate feature dataset.

5. The method according to claim 4, characterized in that, Based on the importance scores of the network traffic features and the shadow features, a subset of the network traffic features are selected to form the candidate feature dataset, including: Obtain the maximum value among the importance scores of all the shadow features; The importance score of each network traffic feature is compared with the maximum value, and the network traffic features with an importance score greater than the maximum value are retained to form the candidate feature dataset.

6. The method according to claim 1, characterized in that, The preset network model is a multilayer perceptron model; the preset network model is trained using the target feature dataset to obtain an abnormal traffic detection model, including: The target feature dataset is input into the multilayer perceptron model for training, and the features in the target feature dataset are weighted and summed using a multi-head attention mechanism to obtain the output result. The loss value of the output result is calculated according to the preset loss function; The weights and biases of each layer in the multilayer perceptron model are optimized based on the loss value and a preset optimization algorithm until convergence or a set number of iterations is reached, thereby obtaining the abnormal traffic detection model.

7. A method for monitoring abnormal flow, characterized in that, include: Obtain the traffic to be detected; The traffic to be detected is input into a preset abnormal traffic detection model to determine whether the traffic to be detected is abnormal; The abnormal traffic detection model is obtained by the method described in any one of claims 1 to 6.

8. A training device for an abnormal traffic detection model, characterized in that, include: The dataset acquisition module is used to acquire network traffic datasets; The network traffic dataset includes multiple network traffic features; The first optimization module is used to eliminate feature redundancy in the network traffic dataset according to the importance of each network traffic feature and form a candidate feature dataset. The second optimization module is used to perform balancing processing on the candidate feature dataset and form the target feature dataset. The training module is used to train a preset network model using the target feature dataset to obtain an abnormal traffic detection model. The second optimization module is used to balance the candidate feature dataset and form the target feature dataset in the following way: For each network traffic feature: determine the distance between the network traffic feature and other network traffic features; determine the network traffic features with a distance less than a preset threshold as nearest neighbor samples; count the types of each nearest neighbor sample and the number of each type, and define the type with the most number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, retain the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, delete the network traffic feature or adjust the preset weight of the network traffic feature, and the remaining network traffic features form the target feature dataset.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the training method of the abnormal traffic detection model according to any one of claims 1 to 6 or the abnormal traffic monitoring method according to claim 7.

Images