Event Message Clustering for Distributed System Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large distributed computing systems, processing and analyzing vast volumes of event messages is challenging due to their unstructured nature, leading to difficulties in detecting operational anomalies and managing system performance effectively.
Innovation Solution
The implementation of an event-message-clustering system that assigns event messages to clusters based on non-parameter tokens, generates parsing functions to extract data, and uses event types to annotate messages, facilitating efficient storage and display of event information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If event messages are processed and stored in their original unstructured format, then all information is preserved, but processing efficiency and analysis capability deteriorate due to the enormous volume and unstructured nature of the data
Solution Approach 1:
The patent segments event messages into structured components including event type, severity level, source component, and parameterized data fields. This segmentation transforms unstructured text into organized data elements that can be processed efficiently while preserving all essential information through systematic categorization and extraction.
Solution Approach 2:
The patent applies parameter changes by converting unstructured event messages into structured formats with defined parameters and data types. This transformation enables efficient querying, filtering, and analysis while maintaining information integrity through standardized parameter representations that capture all essential event characteristics.
2Reliability
If all event messages are stored in detail for later analysis, then complete information is available for diagnostic purposes, but storage requirements and computational overhead increase significantly
Solution Approach 1:
The patent extracts essential diagnostic information from event messages and stores it in a condensed structured format. By taking out only the critical parameters and event characteristics needed for diagnostics, the system maintains diagnostic accuracy while significantly reducing the volume of stored data compared to preserving complete unstructured event logs.
Solution Approach 2:
The patent creates structured copies of event messages that preserve essential diagnostic information in a compact format. These structured representations serve as efficient substitutes for storing complete unstructured event logs, maintaining diagnostic capability while reducing storage requirements through intelligent data representation.
3Extent of automation
If event messages are classified into clusters based on non-parameter tokens, then automated annotation and detection capability improve, but system complexity increases due to clustering algorithms and parsing functions
Solution Approach 1:
The patent performs preliminary clustering and classification of event messages into organized groups based on non-parameter tokens before detailed analysis. This preliminary action creates structured event categories that enable automated annotation and detection, reducing the complexity of subsequent processing by pre-organizing data into meaningful clusters with standardized attributes.
Data Source
AI summary
The current document is directed to methods and systems that process, classify, efficiently store, and display large volumes of event messages generated in modern computing systems. In a disclosed implementation, received event messages are assigned to event-message clusters based on non-parameter tokens identified within the event messages. A parsing function is generated for each cluster that is used to extract data from incoming event messages and to prepare event records from event messages that more efficiently and accessible store event information. The parsing functions also provide an alternative basis for assignment of event messages to clusters. Event types associated with the clusters are used for gathering information from various information sources with which to automatically annotate event messages displayed to system administrators, maintenance personnel, and other users of event messages.