Log Analytics Using Sketching for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale virtualized computing infrastructures, diagnosing and troubleshooting failures and performance issues are challenging due to the overwhelming volume of unstructured log data generated by numerous software and hardware components, making it difficult to identify relevant information within the data.
Innovation Solution
A method for real-time analysis of log messages that classifies messages into types based on content similarity using a sketching algorithm, groups log messages into events, and identifies anomalous events by analyzing patterns in event composition and volume, reducing the complexity of log data for system administrators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If log data from all components is collected and stored in detail for analysis, then diagnostic accuracy and problem identification capability are improved, but data volume and processing complexity increase significantly
Solution Approach 1:
The patent extracts only the essential and relevant features from log data by generating sketches that capture key characteristics of log messages. Instead of storing and analyzing all raw log data, the system extracts salient features into compact sketches, thereby maintaining diagnostic accuracy while dramatically reducing data volume.
Solution Approach 2:
The patent segments log data processing into multiple stages: generating sketches from raw logs, clustering similar sketches into groups, and then analyzing the clustered results. This segmentation allows the system to process data incrementally and focus computational resources on the most relevant portions, reducing overall processing complexity.
2Measurement precision
If all log messages are analyzed in detail to identify patterns and anomalies, then detection precision is improved, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by generating sketches and clustering log messages before conducting detailed anomaly detection. This preprocessing step groups similar log messages together, so that when anomalies are detected, the system only needs to deeply analyze representative samples from each cluster rather than every individual log message, significantly reducing processing time.
Solution Approach 2:
The patent applies partial action by focusing analysis on clustered groups of log messages rather than examining every single message in detail. The sketching and clustering process identifies representative patterns, allowing the system to achieve high detection precision by analyzing a subset of representative data rather than the complete dataset.
3Productivity
If log data is processed and classified in real-time, then responsiveness and early problem detection are improved, but system complexity and computational overhead increase
Solution Approach 1:
The patent replaces complex mechanical-like processing of raw log data with a more efficient computational approach using sketching algorithms and hashing. Instead of traditional text analysis methods, the system uses fingerprint-based sketch generation that is computationally lighter and can be executed in real-time with lower overhead, reducing system complexity while maintaining real-time processing capability.
4Loss of information
If detailed classification of log messages is performed to identify event types, then information quality is improved, but processing complexity and memory requirements increase
Solution Approach 1:
The patent creates simplified copies of log messages in the form of sketches that capture the essential information needed for classification. These sketch copies retain the key identifying features of original log messages while using significantly less memory and computational resources, allowing detailed classification to be performed with reduced processing complexity.
Data Source
AI summary
Large amounts of unstructured log data generated by software and infrastructure components of a computing system are processed and analyzed in real time to identify anomalies and potential problems within the computing system. A log analytics module reduces both the volume and level of detail of log data by first classifying log messages into message types based on their content similarity. The log analytics module may then further reduce data by grouping bursts of log messages into log events. Patterns within these log events, such as the collection and number of different message types that comprise the event, can be used to identify anomalous events.