Anomaly Detection via Time-Decaying Histograms in Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, manual detection of anomalies in log messages from numerous servers is impractical due to high frequency and volume, preventing real-time or near real-time anomaly detection and remediation.
Innovation Solution
A method for computing anomaly scores in real-time or near real-time from event streams, using event-time pairs, property values, and time-decaying histograms to quickly detect anomalies and facilitate remedial actions, which involves processing event-time pairs, computing property values, and updating time-decaying histograms to generate anomaly scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual detection methods are used for anomalies in log messages, then detection accuracy may be maintained, but the system cannot handle high frequency and volume of log messages from numerous servers in real-time
Solution Approach 1:
The patent segments the anomaly detection process into distinct computational stages: log message ingestion, event extraction, histogram computation, anomaly scoring, and alert generation. This segmentation allows each component to be optimized independently and processed in parallel across multiple computing nodes, enabling real-time handling of high-volume log streams from numerous servers without overwhelming a single processing unit.
Solution Approach 2:
The patent replaces manual anomaly detection with an automated computational system that uses algorithms to compute anomaly scores based on historical log data patterns. This substitution of mechanical/manual inspection with automated information processing enables the system to handle high-frequency log messages at scale, transforming the detection capability from human-limited to computationally-limited operations.
2Loss of time
If real-time anomaly detection is implemented, then system downtime can be reduced, but the computational resources and processing complexity increase significantly
Solution Approach 1:
The patent implements partial action by computing anomaly scores only for specific properties of log messages that are most indicative of anomalies, rather than analyzing every aspect of each log message. The system selectively processes event properties and maintains histograms only for relevant metrics, reducing computational overhead while maintaining effective anomaly detection capability to minimize system downtime.
Solution Approach 2:
The patent performs preliminary action by pre-computing historical statistics and maintaining baseline histograms of normal log message patterns before anomalies occur. This pre-established baseline data allows the anomaly detection system to quickly compare current log messages against known normal behavior without requiring intensive real-time computation, thereby reducing computational resource consumption while enabling rapid anomaly identification.
3Measurement precision
If comprehensive log message analysis is performed on all servers, then detection accuracy improves, but the volume of data to be processed becomes unmanageable
Solution Approach 1:
The patent extracts only the essential event properties and characteristics from comprehensive log messages that are most relevant for anomaly detection. Rather than analyzing every field in every log message, the system identifies and extracts key event properties such as error codes, performance metrics, and critical system states, maintaining histograms only for these extracted features. This extraction approach preserves detection accuracy by focusing on discriminative features while dramatically reducing the volume of data that must be processed and stored.
Data Source
AI summary
Event-time pairs are received for a current time slot. Each event-time pair denotes the occurrence of an event at a system by an event type as well as an occurrence time. For each different event type, a property value for the time slot is computed for each different property of a number of different properties, from the event-time pairs having the different event type. For each different property, a time-decaying histogram of identified property values of the different property is updated using the property value computed for the different property for the current time slot. An anomaly score for each identified property value within the time-decaying histogram of each different property is computed to detect occurrence of an anomaly within the system.