Log Event Bin Plotting for Power-Law Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods are inadequate for efficiently identifying anomalies in large volumes of log events from computing devices, particularly in enterprise networks, due to the time-consuming nature of manual inspection and the impracticality of processing vast amounts of data to detect cyber attacks and misconfigurations.
Innovation Solution
A graphical plotting technique is employed to organize devices within log events having a power law-oriented distribution, using first and second metrics to visually identify anomalies by organizing devices over bins and ordering them within each bin based on the second metric, facilitating easier detection of anomalous behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual inspection of log events is performed, then detailed analysis capability is maintained, but time consumption increases significantly
Solution Approach 1:
The patent introduces an automated analysis system that acts as an intermediary between raw log events and human analysts. This system processes log events through multiple stages (filtering, aggregation, anomaly detection) and presents summarized findings to users, thereby maintaining detailed analysis capability while dramatically reducing the time analysts need to spend on manual inspection.
Solution Approach 2:
The analysis process is divided into distinct segments: initial filtering of log events, aggregation of related events, anomaly detection algorithms, and presentation of results. This segmentation allows automated processing of routine tasks while preserving human expertise for complex judgment, resolving the contradiction between automated speed and analytical depth.
2Reliability
If vast amounts of log event data are processed, then detection completeness is improved, but processing complexity increases
Solution Approach 1:
The processing system divides vast log data into manageable segments through filtering and aggregation stages. Each segment is processed independently through anomaly detection algorithms, making the overall complex task tractable while maintaining comprehensive coverage of all log events for complete anomaly detection.
Solution Approach 2:
The system extracts and focuses on only the relevant features and patterns from vast amounts of log data, rather than processing every detail equally. By extracting key indicators of anomalies and concentrating analysis on these extracted features, the system achieves complete detection without being overwhelmed by the full complexity of the raw data.
3Productivity
If automated processing is implemented, then processing speed is improved, but detection precision may deteriorate
Solution Approach 1:
The automated processing system is segmented into multiple specialized stages, each optimized for its specific function. This segmentation allows the system to process data quickly through automated algorithms while preserving precision by having each stage perform its specific task excellently rather than one general-purpose processor attempting everything.
Solution Approach 2:
The system incorporates feedback mechanisms where detection results are continuously refined and validated. Automated processing provides rapid initial detection, and this information feeds back into the system for verification and refinement, ensuring that processing speed does not compromise detection precision.
Data Source
AI summary
For each item represented within log events that have a power law-oriented distribution, first and second metrics for the item are computed based on the log events which pertain to the item. The items are organized over bins according to the first metric. The bins correspond to different ranges of the first metric. For each bin, the items in the bin are ordered according to the second metric. A plot of the bins over which the items have been organized according to the first metric, is graphically displayed, which includes displaying, for each bin, the items in the bin as have been ordered according to the second metric.