Log Event Cluster Analytics via Segmented Dictionary Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for automated analysis of computer-generated event message logs are not well-suited for large-scale systems and networks, as they consume excessive memory and processing time due to the need to compare new event messages with an entire dictionary of clusters, leading to resource overload.
Innovation Solution
The proposed solution involves grouping clusters into families for analysis, storing a portion of the cluster dictionary in transient memory for initial faster analysis, and using a text search engine to identify matches in persistent storage, while distributing analysis across multiple computer nodes to manage large-scale systems efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods compare new event messages with the entire dictionary of clusters, then comprehensive matching accuracy is achieved, but processing time and memory consumption increase excessively
Solution Approach 1:
The patent divides the cluster dictionary into multiple families based on event message types or sources. Instead of comparing against all clusters, the system first identifies the relevant family and then performs matching only within that family, significantly reducing the search space while maintaining matching accuracy.
Solution Approach 2:
The system pre-processes and organizes clusters into families with predefined categories or tags. This preliminary organization allows for rapid family identification before detailed cluster matching, eliminating the need to scan the entire dictionary for every event message.
2Reliability
If traditional methods store the entire cluster dictionary in memory, then complete cluster information is available for analysis, but memory consumption becomes excessive
Solution Approach 1:
The cluster dictionary is segmented into multiple families, with each family stored separately. The system loads only the relevant family into memory during analysis, while other families remain in persistent storage. This segmentation allows complete cluster information to be available across storage while minimizing in-memory requirements.
Solution Approach 2:
The patent introduces a family identification layer as an intermediary between the event message and the cluster dictionary. This intermediary quickly determines which family contains the matching cluster, allowing the system to access only the necessary portion of the dictionary rather than loading everything into memory.
3Measurement precision
If the system analyzes all clusters for every event message, then no matching opportunities are missed, but processing speed decreases
Solution Approach 1:
By segmenting clusters into families and analyzing only the relevant family for each event message, the system maintains complete match detection within the appropriate scope while achieving significant processing speed improvements through reduced search space.
Solution Approach 2:
The system performs preliminary family classification based on event message characteristics before detailed cluster matching. This preliminary action filters out irrelevant families, ensuring no potential matches are missed while dramatically reducing the number of comparisons required.
Data Source
AI summary
A log event cluster analytics management method may involve storing a first portion of an entire cluster dictionary in a transient memory, storing at least a second portion of the entire cluster dictionary in a persistent database and comparing a new log event message to the first portion of the overall cluster dictionary. In response to not assigning the new log event message to any cluster in the first portion of the entire cluster dictionary in the transient memory, selecting a subset of clusters of the at least second portion of the cluster dictionary in the persistent database, comparing the new log event message to a cluster of the selected subset of clusters and assigning the new log event message to the cluster of the selected subset of clusters based upon the comparison.