Log Message Correlation Filtering for Redundant Event Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional log file systems store redundant log messages, consuming storage resources and complicating data analysis, especially when used as inputs for machine learning models, as they often represent multiple messages as a single event.
Innovation Solution
A data-driven method using machine learning and graph theory to generate an uninformative log message filter, identifying and removing redundant messages by creating a correlation graph and applying a filter that separates informative from uninformative log messages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all log messages are stored in the log file system, then complete log data is preserved, but storage resources are consumed and data analysis becomes complex
Solution Approach 1:
The patent extracts redundant log messages from the complete log data set by building a correlation graph and identifying cliques (fully connected subgraphs). Messages that appear in multiple cliques are identified as redundant and removed, keeping only one representative message per event. This extraction process eliminates unnecessary duplication while preserving the essential information needed for analysis.
Solution Approach 2:
The patent changes the parameter of message representation by transforming the log data from a raw collection of messages into a correlation graph structure where nodes represent messages and edges represent co-occurrence relationships. This parameter transformation enables the identification of redundant messages through graph-theoretic operations (finding cliques and their intersections) rather than simple text comparison.
2Loss of information
If all log messages are stored in the log file system, then no information is lost, but data analysis tasks become complicated
Solution Approach 1:
The patent extracts and removes redundant messages that complicate analysis while preserving the unique information content. By identifying messages that appear in multiple cliques of the correlation graph and removing duplicates, the patent simplifies the log data structure, making analysis tasks more manageable without losing essential event information.
Solution Approach 2:
The patent segments the log data into distinct events by identifying cliques in the correlation graph, where each clique represents a potential event. By organizing messages into these segmented event groups and removing redundant representations, the patent creates a structured, simplified data format that is more amenable to analysis while preserving complete event information.
3Loss of information
If multiple log messages represent a single system event, then detailed information is captured, but the concept of single-event-single-message is violated
Solution Approach 1:
The patent extracts one representative message from each set of redundant messages that represent the same event. By identifying cliques in the correlation graph and removing duplicate messages across these cliques, the patent achieves a one-to-one correspondence between events and messages, eliminating the violation of the single-event-single-message concept while preserving detailed event information through the selected representative messages.
Data Source
AI summary
Techniques are provided for data-driven reduction of log message data. An exemplary method comprises: obtaining log files and user-specified configuration parameters, wherein the log files each comprise one or more log messages; generating an event count matrix indicating a number of times each of a plurality of unique messages appeared in a given log file of the log files; generating a correlation graph by inserting similar messages with a mutual undirected edge, wherein similar messages are identified based on a predefined similarity measure; extracting redundant messages from the correlation graph by selecting log messages for inclusion in an uninformative log message filter from sub-graphs of the correlation graph in which any two nodes are connected together, except those log messages satisfying a predefined message frequency criteria; and identifying one or more redundant messages using the uninformative log message filter. The uninformative log message filter is optionally applied to real-time log messages and/or existing file systems.