Anomaly Detection in Data Communications Using Interpretable AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current unsupervised learning models in anomaly detection for communication networks lack interpretability, making it difficult to understand and explain the decisions made by AI systems, which is crucial for transparency and trustworthiness in cybersecurity applications.
Innovation Solution
A method that uses an unsupervised AI learning model incorporating local, specific, and counterfactual explanations to detect anomalies in communication data, employing a non-parametric statistical model with a radial basis function kernel for anomaly identification, and visualizing results through a graphical user interface, allowing for understandable explanations of detected anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If unsupervised learning models are used for anomaly detection in communication networks, then detection capability is improved, but interpretability and transparency deteriorate
Solution Approach 1:
The patent introduces an explanation generation module as an intermediary between the unsupervised learning model and the user. This module translates the black-box model outputs into human-understandable explanations, including local explanations for individual anomalies and global explanations for overall patterns, thereby maintaining detection capability while improving interpretability
Solution Approach 2:
The explanation system is segmented into multiple components: local explanations for individual anomaly points, global explanations for overall data patterns, and counterfactual explanations showing what would make data normal. This segmentation allows different types of interpretability to address different user needs without compromising detection performance
2Measurement precision
If non-parametric statistical models with RBF kernel are used, then detection precision is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-processing the communication data to extract relevant features and normalize them before applying the RBF kernel. This preliminary feature extraction reduces the dimensionality and complexity of the input data, allowing the non-parametric model to achieve high precision with reduced computational burden
Solution Approach 2:
The system dynamically adjusts parameters such as the RBF kernel bandwidth and decision threshold based on the characteristics of the communication data. By optimizing these parameters, the model achieves high detection precision while avoiding excessive computational complexity through adaptive parameter selection rather than exhaustive computation
3Loss of information
If local explanations are generated for each anomaly point, then transparency is improved, but processing time increases
Solution Approach 1:
The patent extracts only the most relevant features and explanation elements for each anomaly point rather than generating comprehensive explanations for all possible factors. This selective extraction provides sufficient transparency for understanding anomalies while significantly reducing the processing time required to generate explanations
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for detecting anomalies in data communications comprising: obtaining a set of data points, wherein each data point is defined by a datum that is a continuous variable and one or more categories of the datum that are discrete variables; preprocessing (4) the set of data points to introduce them into a non-parametric statistical model (31); performing a grid search (5, 6) to obtain the hyperparameters to train (7) the model; applying an RBF kernel function to identify anomalies (10) using a nonlinear decision threshold and calculating a distance from the value of the continuous variable to the value of the decision threshold according to the hyperparameters for each combination of categories, to obtain upper and lower limits (8, 14); displaying (11) the results comparing the limits obtained (8, 14) with the distance calculated for each datum and combination of categories.