Anomaly Detection in Network Logs via Entropy and Sentiment Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In network environments, diagnosing issues from unstructured log data is challenging due to varying formats and semantics, and existing rule-based approaches struggle to detect unknown anomalies effectively, often leading to false positives and inefficient troubleshooting.
Innovation Solution
A real-time anomaly detection system using an unsupervised machine learning model that learns normal network behavior from log entries, identifies anomalous events, and filters noteworthy ones based on entropy and sentiment analysis, reducing false positives and improving reporting efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If rule-based approaches are used for anomaly detection, then the system can detect known anomalies, but it fails to detect unknown anomalies and produces false positives
Solution Approach 1:
The patent replaces rule-based mechanical anomaly detection with unsupervised machine learning models that automatically learn patterns from log data. The system uses sequence-to-sequence models and entropy calculations to detect anomalies without predefined rules, enabling both high reliability for known anomalies and adaptability for unknown anomaly types.
Solution Approach 2:
The patent transforms the anomaly detection approach by changing from fixed rule parameters to dynamic statistical parameters. It uses entropy calculations based on log sequence probability distributions to adaptively determine anomaly thresholds, allowing the system to adjust to changing network conditions while maintaining detection accuracy.
2Measurement precision
If comprehensive log analysis is performed to improve anomaly detection, then detection accuracy improves, but system complexity and processing time increase
Solution Approach 1:
The patent extracts only the most relevant features from log data for anomaly detection. It focuses on sequence probability distributions and entropy calculations rather than analyzing all log attributes, thereby maintaining high detection precision while reducing system complexity and processing requirements.
Solution Approach 2:
The patent segments the anomaly detection process into distinct components: log sequence modeling using neural networks, entropy calculation for anomaly scoring, and threshold-based filtering. This modular approach improves precision through specialized processing while managing overall system complexity through clear separation of functions.
3Reliability
If all anomalous events are reported to improve completeness, then all anomalies are captured, but false positives increase and reporting efficiency decreases
Solution Approach 1:
The patent uses entropy as a continuous parameter to score anomalous events rather than using binary anomaly classification. By calculating the probability distribution of log sequences and determining entropy values, the system can prioritize reports based on anomaly severity, ensuring complete capture of significant anomalies while filtering out low-impact false positives to improve reporting efficiency.
Data Source
AI summary
Anomaly detection and filtering based on system logs is provided via receiving, at a first time, a first sequence of log entries from a networked system; generating, based on previously observed log entries including the first sequence of log entries, predicted log entries for a second time; receiving, at the second time, a second sequence of log entries from the networked system; determining whether the second sequence is anomalous based on comparing the second sequence with the predicted log entries; in response to determining that the second sequence is anomalous, determining whether the second sequence is noteworthy based on a function entropy of the first sequence and a sentiment polarity of the second sequence; and in response to determining that the second sequence is noteworthy, generating an anomaly report that includes the second sequence and a root cause.