Log Pattern Detection for Runtime Log Flooding Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex computing systems experience log flooding, leading to network bandwidth saturation, storage overload, and system degradation due to sudden spikes in log data, which can cause system-wide crashes and data loss.
Innovation Solution
A log management system that detects and addresses excessive log patterns by tracking cumulative attributes, using machine learning and pattern recognition to filter and remove instances of problematic patterns, thereby reducing the amount of stored log data and preventing resource oversaturation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log data is recorded continuously to monitor system behavior, then system reliability and diagnostic capability are improved, but storage resources are overwhelmed and network bandwidth is saturated due to log flooding
Solution Approach 1:
The patent extracts and removes duplicate log entries from the log stream by comparing incoming logs against a repository of previously seen logs. When duplicates are detected, they are filtered out before being written to storage or transmitted over the network, thereby reducing log data volume while preserving unique diagnostic information
Solution Approach 2:
The system performs preliminary detection and classification of log patterns before full processing. By identifying duplicate patterns early in the log stream using pattern matching and fingerprinting techniques, the system prevents redundant data from consuming storage and network resources in the first place
2Loss of information
If all log data is stored for analysis, then complete diagnostic information is preserved, but system performance degrades due to storage overload and increased I/O operations
Solution Approach 1:
The patent extracts only unique log entries from the continuous log stream, removing duplicates before storage. This significantly reduces the amount of data written to disk, decreasing I/O operations and storage overhead while maintaining complete diagnostic information about unique system events and errors
Solution Approach 2:
The system changes the parameter of log data uniqueness by using fingerprinting and pattern matching to identify duplicate entries. By transforming the log processing approach from storing all entries to storing only unique entries based on computed fingerprints, the system reduces storage I/O while preserving diagnostic value
3Reliability
If log transmission is increased to capture all events, then monitoring coverage is improved, but network bandwidth is saturated causing systemwide crashes
Solution Approach 1:
The patent extracts duplicate log entries from the transmission stream by comparing against a repository of known logs. Only unique log entries are transmitted over the network, dramatically reducing network bandwidth consumption while maintaining complete monitoring coverage of unique system events and conditions
Solution Approach 2:
The system performs preliminary duplicate detection and filtering before log transmission. By computing fingerprints and comparing incoming logs against the repository in advance, the system prevents redundant data from occupying network bandwidth, ensuring efficient use of network resources while maintaining monitoring completeness
Data Source
AI summary
Embodiments described herein include a log management system that detects and addresses excessive log patterns at runtime. In some embodiments, the log management system tracks cumulative attributes associated with patterns that repeat within a set monitoring files, such as a set of log or trace records. The cumulative attributes that are monitored may include the cumulative storage size and/or cumulative count of patterns detected in the monitoring files. The log management system may determine whether a triggering condition is satisfied for initiating a responsive action to reduce the amount of log data that is transmitted and/or stored. If the triggering condition is satisfied, then the log management system may remove one or more instances of the pattern from the monitoring files and/or perform other actions to reduce the overhead of logging operations.