Anomaly Detection Algorithms for SIEM Log Prioritization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Security Information and Event Management (SIEM) systems face high latency and degraded performance due to the large volume of logs ingested daily, with nearly 95-98% being benign, leading to inadequate results and user dissatisfaction, as they process excessive logs per second and offenses per month, requiring inefficient resource allocation and inadequate rule tuning.
Innovation Solution
Implementing a method that uses multiple anomaly detection algorithms (Isolation Forest, Self-Organizing Map, and DBSCAN) to filter and prioritize logs in real-time, identifying and routing only high-priority anomalous logs to the SIEM, while normal logs are processed by low-cost analytics, thereby reducing the workload and improving system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If SIEM systems process all incoming logs, then comprehensive security monitoring is achieved, but system latency increases and performance degrades
Solution Approach 1:
The log processing system is segmented into two distinct pathways: a fast path using anomaly detection algorithms for real-time filtering, and a slow path using comprehensive SIEM analysis for deep security investigation. This segmentation allows the system to maintain comprehensive monitoring capability while reducing latency for the majority of benign logs through the fast path.
Solution Approach 2:
The patent extracts and removes benign logs from the main SIEM processing pipeline by applying anomaly detection algorithms that identify and filter out normal logs before they reach the SIEM system. This extraction reduces the workload on SIEM and eliminates latency caused by processing unnecessary logs, while still maintaining comprehensive security monitoring for anomalous events.
2Reliability
If SIEM systems ingest large volumes of logs, then complete security analysis is possible, but resource allocation becomes inefficient
Solution Approach 1:
The system performs preliminary filtering of logs using anomaly detection algorithms before logs enter the SIEM system. This preliminary action identifies and removes benign logs in advance, so that only potentially security-relevant logs are subjected to resource-intensive SIEM analysis, thereby improving resource allocation efficiency while maintaining complete security analysis capability for relevant events.
Solution Approach 2:
Different processing qualities are applied to different logs based on their characteristics: benign logs receive minimal processing through anomaly detection filtering, while anomalous logs receive comprehensive analysis through SIEM. This local quality approach optimizes resource allocation by applying appropriate levels of analysis to each log based on its security relevance.
3Reliability
If all logs are processed by SIEM, then no anomalies are missed, but analyst efficiency decreases due to high volume of benign logs
Solution Approach 1:
The system extracts benign logs from the processing stream using anomaly detection algorithms, removing them before they reach analysts. This extraction ensures that analysts only receive and review anomalous logs that require human attention, maintaining anomaly detection completeness while dramatically improving analyst efficiency by eliminating the burden of reviewing thousands of benign logs.
Solution Approach 2:
The anomaly detection algorithm acts as an intermediary between log generation and analyst review. This intermediary automatically filters and prioritizes logs, presenting only relevant anomalies to analysts. The intermediary maintains detection completeness by ensuring all anomalies are captured while improving efficiency by removing benign logs from the analyst workflow.
Data Source
AI summary
A method identifies and prioritizes anomalies in received monitoring logs from an endpoint log source. One or more processors identify anomalies in the monitoring logs by applying a plurality of disparate types of anomaly detection algorithms to the monitoring logs, and then determine a likelihood that the identified anomalies are anomalous based on outputs of the plurality of disparate types of anomaly detection algorithms. The processor(s) then prioritize the monitoring logs based on the likelihood that the identified anomalies are actually anomalous, and send prioritized monitoring logs that exceed a priority level to a security information and event management system (SIEM).