Network Anomaly Detection via Graph Path Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing large and highly connected master network graphs from log files to detect anomalies is computationally expensive due to exponential time complexity and memory usage, making it challenging to identify cyber security threats efficiently.
Innovation Solution
A network sampling based path decomposition and anomaly detection system that samples the master network graph to produce a representative graph with a reduced number of paths, selected by probability, allowing for uniform sampling and preserving walk probability distributions, thereby reducing analysis time and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exhaustive analysis of all paths in the master network graph is performed to ensure complete anomaly detection, then measurement precision is improved, but loss of time and computing resource utilization deteriorate
Solution Approach 1:
The patent segments the master network graph into multiple subgraphs by applying community detection algorithms. This divides the large-scale graph analysis into smaller, more manageable subgraph analyses, reducing the computational complexity from exponential to polynomial time while maintaining anomaly detection effectiveness through hierarchical processing.
Solution Approach 2:
The patent implements partial action by analyzing only the most relevant subgraphs and paths rather than performing exhaustive analysis of all possible paths. By identifying and focusing on high-probability anomaly paths and suspicious subgraphs, the system achieves effective anomaly detection without the prohibitive computational cost of complete enumeration.
2Measurement precision
If exhaustive analysis of all paths in the master network graph is performed to ensure complete anomaly detection, then measurement precision is improved, but computing resource utilization deteriorates
Solution Approach 1:
The patent segments the master network graph into multiple subgraphs by applying community detection algorithms. This divides the large-scale graph analysis into smaller, more manageable subgraph analyses, reducing the computational complexity from exponential to polynomial time while maintaining anomaly detection effectiveness through hierarchical processing.
Solution Approach 2:
The patent creates simplified copies of the master network graph in the form of subgraphs that retain only the essential structural and behavioral characteristics needed for anomaly detection. These subgraph copies enable efficient analysis without requiring resources proportional to the full master graph.
3Productivity
If sampling is applied to reduce the number of paths analyzed, then productivity is improved, but measurement precision deteriorates
Solution Approach 1:
The patent implements partial action by analyzing only the most relevant subgraphs and paths rather than performing exhaustive analysis of all possible paths. By identifying and focusing on high-probability anomaly paths and suspicious subgraphs, the system achieves effective anomaly detection without the prohibitive computational cost of complete enumeration.
Solution Approach 2:
The patent changes the parameters of analysis by transitioning from uniform random sampling to probability-proportional-to-size (PPS) sampling. This parameter change ensures that larger, more significant subgraphs have higher selection probabilities, maintaining measurement precision while improving productivity through targeted sampling of the most relevant graph portions.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
According to examples, network sampling based path decomposition and anomaly detection may include evaluating computer-generated log file data to generate a master network graph that specifies known events and transitions between the known events, and decomposing the master network graph to generate a representative network graph that includes a reduced number of paths of the master network graph. A source may be monitored to determine a cyber security threat by receiving incoming log file data related to the source, comparing the incoming log file data related to the source to the representative network graph, and determining, based on the comparison of the incoming log file data related to the source to the representative network graph, an anomaly in the representative network graph. Further, based on the monitoring, a report indicative of the cyber security threat may be generated based on the anomaly in the representative network graph.