Graph ML Explainability for Invalid Network Traffic Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models for detecting invalid network traffic, such as from botnets, fail to adequately capture interactions between agents and resources, and they do not leverage complex dependencies between multiple agents that may reveal malicious coordination.
Innovation Solution
The use of graph machine learning approaches that model network traffic data as a bipartite heterogeneous graph, incorporating graph embedding, clustering, and unsupervised training techniques to explain why certain network traffic is suspicious, without requiring labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rule-based systems are used to identify bots, then precision is high in simple scenarios, but the system cannot detect sophisticated bots and depends on domain expertise
Solution Approach 1:
The patent replaces rule-based mechanical detection systems with machine learning models that automatically learn detection patterns from data. The system uses supervised learning algorithms to train detectors on labeled bot traffic data, enabling automatic adaptation to sophisticated bot variants without requiring manual rule updates or domain expertise for each new threat type.
Solution Approach 2:
The patent transforms the detection approach by changing from fixed rule parameters to learned model parameters. The machine learning models automatically adjust their internal parameters (weights, biases, thresholds) based on training data, allowing the system to adapt to evolving bot techniques while maintaining high precision through data-driven parameter optimization.
2Adaptability or versatility
If machine learning models are used to classify agents, then detection capability improves, but the models do not adequately capture interactions between agents and resources
Solution Approach 1:
The patent adds a new dimension to the detection space by incorporating interaction features between agents and resources. The model processes not only agent characteristics but also their interaction patterns with resources, creating a multi-dimensional feature space that captures the complex relationships. This dimensional expansion allows the model to detect bots based on their interaction behavior while preserving information about agent-resource relationships.
3Extent of automation
If traditional machine learning models are used, then classification is possible, but they fail to leverage complex dependencies between multiple agents that may reveal malicious coordination
Solution Approach 1:
The patent segments the analysis into multiple processing stages: individual agent feature extraction, interaction pattern detection, and coordinated behavior analysis. By dividing the detection process into segments that handle different aspects (single agent characteristics, pairwise interactions, group coordination), the model can automatically process complex dependencies while maintaining computational efficiency and information preservation.
4Measurement precision
If labeled data is used for training, then model accuracy can be improved, but labeled data is expensive and difficult to obtain
Solution Approach 1:
The patent performs preliminary unsupervised pre-training on large amounts of unlabeled network traffic data to learn general patterns and representations. This preliminary action allows the model to acquire useful knowledge without requiring labeled data, and then fine-tuning with a small amount of labeled data achieves high accuracy while minimizing the cost and effort of data labeling.
Data Source
AI summary
Herein are graph machine learning explainability (MLX) techniques for invalid traffic detection. In an embodiment, a computer generates a graph that contains: a) domain vertices that represent network domains that received requests and b) address vertices that respectively represent network addresses from which the requests originated. Based on the graph, domain embeddings are generated that respectively encode the domain vertices. Based on the domain embeddings, multidomain embeddings are generated that respectively encode the network addresses. The multidomain embeddings are organized into multiple clusters of multidomain embeddings. A particular cluster is detected as suspicious. In an embodiment, an unsupervised trained graph model generates the multidomain embeddings. Based on the clusters of multidomain embeddings, feature importances are unsupervised trained. Based on the feature importances, an explanation is automatically generated for why an object is or is not suspicious. The explained object may be a cluster or other batch of network addresses or a single network address.