Event Message Clustering via Non-Parameter Token Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large distributed computing systems, processing and analyzing vast volumes of event messages is challenging due to their unstructured nature, leading to difficulties in identifying relevant information for system management, security, and diagnostics, and requires efficient storage solutions to handle the massive data generated daily.
Innovation Solution
The system normalizes event messages to identify non-parameter tokens, computes a metric for each message, and assigns it to a type-associated cluster, allowing for dynamic clustering, splitting, and merging to refine event-message typing, enabling efficient storage and analysis by transforming event logs into typed event records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If event messages are stored in their original unstructured form, then all information is preserved, but storage space requirements increase significantly and analysis becomes difficult
Solution Approach 1:
The event message is segmented into structured components: event type identifier and parameter data. This segmentation allows the system to separate the common template (stored once) from the variable parameters (stored with each event), reducing redundant storage while preserving all information.
Solution Approach 2:
The system transforms unstructured event messages into a parameterized format by identifying and extracting parameter tokens. This parameterization enables compact storage where only varying parameters need to be stored repeatedly, while the fixed template structure is stored once per event type.
2Productivity
If event messages are processed and classified into clusters, then analysis efficiency improves, but processing complexity increases
Solution Approach 1:
The system performs preliminary classification by computing a cluster identifier for each event type before full analysis. This preliminary action groups similar events together, enabling more efficient subsequent analysis while keeping the processing complexity manageable through incremental refinement.
Solution Approach 2:
The clustering structure is dynamic and adaptive. The system can refine clusters by splitting them or merging them based on analysis needs, allowing the processing complexity to adapt to the specific requirements of different analysis scenarios while maintaining high efficiency.
3Measurement precision
If dynamic clustering is implemented to refine event typing, then classification accuracy improves, but computational overhead increases
Solution Approach 1:
The system implements partial clustering refinement by dynamically splitting or merging clusters only when necessary to achieve the required classification accuracy. This selective refinement approach maintains high classification precision while minimizing the computational overhead associated with continuous full-scale clustering operations.
Data Source
AI summary
The current document is directed to methods and systems for processing, classifying, and efficiently storing large volumes of event messages generated in modern computing systems. In a disclosed implementation, received event messages are normalized to identify non-parameter tokens within the event messages. The non-parameter event tokens are used to compute a metric for each event message. The metrics are used, in turn, to identify a type-associated cluster to which to assign each received event message. The type-associated clusters are created dynamically as streams of event messages are processed. The type-associated clusters may be dynamically split and merged to refine event-message typing.