Hybrid ML Anomaly Detection for Log Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of computer system logs in modern computing environments makes manual anomaly detection impractical, with existing Security Information and Event Management (SIEM) systems generating high false positives and requiring significant manual tuning and labeling efforts from human security analysts.
Innovation Solution
A hybrid machine learning approach combining unsupervised and supervised learning models to detect anomalies in log data, where unsupervised models generate initial anomaly scores, and semi-supervised models propagate labels based on similarity analysis, reducing manual effort and improving accuracy through dynamic weighting of model outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual review of computer system logs is performed by human security analysts, then anomaly detection can be conducted, but the process becomes impractical due to explosive growth of log volumes
Solution Approach 1:
The patent replaces the mechanical manual review process with an automated machine learning system. The hybrid ML anomaly detector uses unsupervised learning models to automatically analyze log entries, generate anomaly scores, and classify anomalies without human intervention, thereby maintaining detection accuracy while scaling to handle explosive log volume growth
Solution Approach 2:
The patent introduces machine learning models as an intermediary between raw log data and security analysts. The unsupervised models process logs and generate preliminary anomaly assessments, which then feed into the semi-supervised models for refined classification, reducing the burden on human analysts while preserving detection capabilities
2Extent of automation
If SIEM computing systems apply rules engine and search criteria for anomaly detection, then automated detection is achieved, but a large number of false positives are generated requiring human review
Solution Approach 1:
The patent changes the detection parameters from binary rule-based matching to continuous anomaly scoring. The unsupervised models generate nuanced anomaly scores that reflect the degree of abnormality, allowing for more reliable automated detection with fewer false positives by thresholding these scores rather than relying on rigid rule criteria
Solution Approach 2:
The patent performs preliminary anomaly scoring using unsupervised models before final classification. This preliminary action filters and prioritizes log entries, so that only those with significant anomaly scores proceed to semi-supervised classification, reducing false positives in the final detection output
3Ease of operation
If rules-based SIEM systems are used for anomaly detection, then manual tuning and codification by security analysts is required, but this increases device complexity and manual effort
Solution Approach 1:
The patent enables the system to self-configure through unsupervised learning. The models automatically learn patterns and generate anomaly detection logic from the log data itself without requiring security analysts to manually codify rules, thereby simplifying operation while reducing the apparent complexity through automation
4Measurement precision
If supervised machine learning models are trained with manually labeled data, then detection accuracy improves, but the extensive manual labeling effort becomes a bottleneck
Solution Approach 1:
The patent performs preliminary anomaly scoring using unsupervised models to automatically generate initial labels for training data. This preliminary labeling action provides sufficient training signals for semi-supervised models, dramatically reducing the manual labeling effort required while maintaining detection accuracy
Solution Approach 2:
The patent applies different labeling strategies to different portions of the data. High-confidence anomalies identified by unsupervised models receive automatic labels, while uncertain cases are reserved for manual review, optimizing the balance between automation and manual effort in the training process
Data Source
AI summary
Mechanisms are provided to implement a hybrid machine learning (ML) anomaly detector comprising an ensemble of unsupervised ML models and a semi-supervised ML model. The ensemble of unsupervised ML models are executed on log data to generate, for each entry in the log data, a predicted anomaly score and corresponding anomaly classification label of the entry. A partially labeled dataset is generated based on a selected subset of entries and other unlabeled log data in the log data. A similarity analysis of the unlabeled log data with entries in the selected subset of entries is performed and anomaly classification labels of the selected subset of entries are propagated to the other unlabeled log data based on the similarity analysis.