Log File Sampling for Anomaly Detection in Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large distributed computing systems, identifying relevant information from enormous event logs to detect operational anomalies and manage systems effectively is challenging due to the volume and complexity of data generated.
Innovation Solution
The method involves log file sampling, where event messages are collected in intervals to calculate metric values representing behavioral changes, with alerts generated when differences exceed thresholds, using techniques like Jensen-Shannon divergence to compare probability distributions of event types across time intervals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complete event logs are analyzed to detect anomalies, then detection accuracy is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the event log analysis by collecting event messages in discrete time intervals (e.g., 5-minute intervals) rather than processing the entire log continuously. This temporal segmentation allows the system to analyze manageable portions of data while maintaining detection accuracy through multiple sampling points throughout the monitoring period.
Solution Approach 2:
The patent applies partial action by sampling only a portion of event messages from the complete log file. Instead of analyzing every event message, the system collects samples at multiple time intervals, which provides sufficient statistical representation to detect anomalies while significantly reducing processing time and computational resources required.
2Reliability
If event messages are collected continuously to capture all behavioral changes, then detection completeness is improved, but system complexity and resource consumption increase
Solution Approach 1:
The patent implements periodic action by collecting event messages at regular time intervals rather than continuously. The system samples events at predetermined intervals (e.g., every 5 minutes) and compares these periodic samples to detect changes in system behavior, which maintains detection completeness while simplifying the collection mechanism and reducing resource consumption.
Solution Approach 2:
The patent uses feedback mechanisms by comparing event message samples from different time intervals to identify behavioral changes. The system calculates metric values representing differences between samples and generates alerts when changes exceed thresholds, creating a feedback loop that maintains detection reliability without requiring continuous complex processing.
3Measurement precision
If detailed analysis of all event messages is performed, then anomaly detection precision is improved, but computational resources and processing overhead increase
Solution Approach 1:
The patent applies partial action by analyzing only sampled event messages rather than the complete log. The system collects samples at multiple time intervals and performs detailed analysis only on these samples, which provides sufficient precision for anomaly detection while significantly reducing computational resource consumption compared to analyzing every event message in detail.
Solution Approach 2:
The patent segments the event log into multiple time-based samples, allowing the system to perform detailed analysis on manageable portions of data. This segmentation enables precise anomaly detection through statistical comparison of samples while reducing the computational burden of processing the entire log file at once.
Data Source
AI summary
Methods and systems that detect computer system anomalies based on log file sampling are described. Computers systems generate log files that record various types of operating system and software run events in event messages. For each computer system, a sample of event messages are collected in a first time interval and a sample of event messages are collected in a recent second time interval. Methods calculate a difference between the event messages collected in the first and second time intervals. When the difference is greater than a threshold, an alert is generated. The process of repeatedly collecting a sample of event messages in a recent time interval, calculating a difference between the event messages collected in the recent and previous time intervals, comparing the difference to the threshold, and generating an alert when the threshold is violated may be executed for each computer system of a cluster of computer systems.