Log Anomaly Detection via Dynamic Sample Set Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log anomaly detection methods face challenges in achieving high accuracy due to the 'curse of dimensionality' and model drift when processing high-dimensional or dynamically changing log data.
Innovation Solution
A log anomaly detection method that involves sampling log data, deleting samples with low availability, and determining anomaly based on the probability of log events falling into the sample set, ensuring real-time updating and dynamic adaptation to log data changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If machine learning methods are adopted to perform anomaly detection on log data, then detection capability for new and unknown threats is improved, but the phenomenon of 'curse of dimensionality' and model drift occurs leading to reduced accuracy
Solution Approach 1:
The patent extracts only the necessary information from log data by deleting samples that do not meet predefined conditions (such as samples with missing critical fields or outliers). This extraction process creates a cleaned, relevant sample set that reduces dimensionality while maintaining the essential characteristics needed for accurate anomaly detection, thereby mitigating the curse of dimensionality without losing detection capability for new threats.
Solution Approach 2:
The patent implements dynamic sample selection by continuously updating the sample set based on predefined conditions and deleting samples that no longer meet these conditions. This dynamic approach allows the system to adapt to changing log data characteristics over time, preventing model drift while maintaining high detection accuracy for both known and unknown threats.
2Reliability
If rules and signatures are used for anomaly detection, then known threats can be detected, but accuracy is low for new and unknown threats and cost of maintaining rules increases
Solution Approach 1:
The patent enables the system to automatically learn from log data patterns and build its own detection capabilities through unsupervised learning. By continuously analyzing log data and identifying anomalies without requiring manual rule creation, the system self-updates its detection models, eliminating the need for manual rule maintenance while gaining the ability to detect both known and unknown threats effectively.
3Loss of information
If high-dimensional log data is processed directly, then comprehensive information is captured, but similarity computations and model training become difficult due to 'curse of dimensionality'
Solution Approach 1:
The patent extracts and selects only the most relevant features from high-dimensional log data by deleting samples that do not meet predefined conditions. This extraction reduces the dimensionality of the data while preserving the essential information needed for anomaly detection, making similarity computations and model training feasible without losing critical insights.
Solution Approach 2:
The patent segments the log data processing into distinct stages: data collection, sample selection based on predefined conditions, and anomaly detection. This segmentation allows the system to handle high-dimensional data systematically by processing it in manageable chunks, reducing computational complexity while maintaining comprehensive information capture.
Data Source
AI summary
The present disclosure relates to a log anomaly detection method and apparatus, a computer device, and a storage medium. By sampling first log data to obtain a sample set of the first log data, a sample with low availability is deleted in the sampling process; the sampling of the first log data is stopped in response to the sampling completeness being high, and a sample set of the first log data is outputted; and whether second log data is anomalous data is determined based on the probability of log events from the second log data falling into the sample set of the first log data. Due to real-time updating of the sample set based on the sample availability during sampling, dynamic adaption to changes of the log data can be realized, thereby improving the anomaly detection accuracy and detection efficiency of the dynamic log data.