Storage System Log Pattern Detection for Overlogging Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Excessive logging in storage systems leads to resource consumption, suppression of real anomalous logs, large log file sizes, increased bandwidth and storage requirements, and prolonged troubleshooting times, hindering system performance and increasing costs.
Innovation Solution
A computer-implemented method processes log files using multiple threads to identify log patterns, filter out excessive logging, and generate alerts for potential overlogging issues, allowing for data collection and report generation to remediate the issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If logging is designed as system-wide and shared amongst different components, then comprehensive system monitoring is achieved, but resource consumption increases and system performance deteriorates
Solution Approach 1:
The patent segments the monolithic logging system into component-specific logging instances. Each storage component (e.g., disk array, controller) maintains its own logging subsystem with independent configuration, allowing granular control over logging intensity and content for each component while maintaining comprehensive system-wide monitoring capability.
2Loss of information
If storage products print massive amounts of log lines to capture all system activities, then complete troubleshooting information is available, but log file sizes become petabytes large requiring excessive bandwidth and disk space
Solution Approach 1:
The patent implements local quality by allowing different logging policies and retention periods for different components and log types. Critical error logs from storage controllers are retained with high fidelity and long duration, while routine operational logs from less critical components use compression and shorter retention, optimizing the balance between information completeness and storage requirements.
3Reliability
If excessive logging is performed to capture all potential issues, then no anomalous log is missed, but real anomalous logs are suppressed and troubleshooting time increases
Solution Approach 1:
The patent implements feedback mechanisms where logging behavior is dynamically adjusted based on system state and historical analysis. The system learns from patterns in log data and automatically adjusts logging intensity, applying higher logging rates during periods of system instability or anomaly detection while reducing logging during normal operation, thereby preventing log suppression while minimizing troubleshooting overhead.
4Reliability
If logging resources are shared amongst multiple components, then system-wide monitoring is achieved, but components race for logging resources causing performance degradation
Solution Approach 1:
The patent segments logging resources into component-specific pools, with each storage component allocated dedicated logging buffers, file handles, and processing threads. This eliminates resource contention and racing conditions while maintaining system-wide monitoring coverage through coordinated logging instances that share metadata and correlation information without competing for underlying resources.
Data Source
AI summary
A method, computer program product, and computing system for processing a plurality of log files from one or more storage systems. A log pattern concerning at least two or more log files is identified by processing each log file from the plurality of log files using one or more processing threads. A potential overlogging issue associated with the log pattern is identified.