Alert Clustering Analysis for Overdetection Review in Network Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques fail to effectively group alerts of communication with similar features, leading to unclear group features and excessive operator workload in determining overdetection, even when grouping by data amount.
Innovation Solution
An analysis server accumulates alerts, performs clustering using communication features, determines cluster homogeneity, and outputs results to reduce overdetection determination workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If alerts are grouped using conventional techniques by category variables such as terminal device ID, protocol, and port number, then the number of alerts presented to operators is reduced, but alerts with the same communication features but different data amounts remain ungrouped, requiring separate operator review
Solution Approach 1:
The patent segments alert grouping into multiple hierarchical levels: first by category variables (protocol, port number), then by communication features (destination IP, destination port), and finally by data amount characteristics. This multi-level segmentation allows alerts with identical communication features to be grouped together regardless of data amount variations, reducing the number of separate reviews operators must perform while maintaining clear group characteristics at each segmentation level.
2Measurement precision
If alerts are grouped by data amount, then alerts with similar data characteristics can be identified, but when the number of feature amounts increases, the features of each group become unclear
Solution Approach 1:
The patent applies local quality by assigning different weighting factors to different feature amounts based on their discriminative value. Communication features such as destination IP address and destination port number are given higher weights because they fundamentally characterize the communication type, while data amount features receive lower weights. This localized quality assignment ensures that groups remain clear and distinguishable even when multiple features are considered, preventing group特征 blurring.
Solution Approach 2:
The patent transforms the grouping problem from a single-dimensional data amount comparison into a multi-dimensional feature space analysis. By considering category variables, communication features, and data amounts as dimensions in a feature space and using distance calculations in this space, the system can effectively group alerts while maintaining clear group boundaries even with multiple features, avoiding the complexity issue of traditional multi-feature grouping.
3Reliability
If a model indicating normal communication features is used for abnormality detection, then normal communication patterns can be identified, but new normal communication is detected as abnormal (overdetection)
Solution Approach 1:
The patent implements dynamic model updates by continuously learning from newly identified normal communication patterns. When operators review clustered alerts and confirm certain communication patterns as normal, the system incorporates these patterns into the baseline model through continuous learning. This dynamic adaptation allows the model to evolve and recognize new normal communication types, reducing overdetection while maintaining reliability for detecting actual abnormalities.
Solution Approach 2:
The patent establishes a feedback loop where operator decisions regarding clustered alerts are fed back into the system. When operators determine that certain alerts represent normal communication, this information is used to update the baseline model. The feedback mechanism enables the system to learn from human expertise and adjust its detection criteria, thereby reducing overdetection of new normal communication while maintaining accurate abnormality detection.
Data Source
AI summary
An analysis server accumulates an alert of communication determined not to be normal communication on the basis of a model indicating a feature of normal communication in a storage unit. Then, the analysis server performs clustering of alerts obtained by excluding an alert having a different category variable from communication data used for learning of the model from the accumulated alerts by using a feature amount of communication included in the alert. Thereafter, the analysis server determines, for each cluster generated by clustering, whether or not the cluster includes the same type of alert. Then, the analysis server outputs a result of clustering and a determination result as to whether or not each cluster includes the same type of alert.