System Log Preprocessing for Duplicate Error Condensation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex computing systems generate a large number of error events quickly, leading to redundant information in log files that causes frequent rollover, difficulty in understanding, and performance impacts, with users often unaware due to disabled notifications.
Innovation Solution
A method that detects errors over a specified period, stores them in a memory buffer, preprocesses to condense and remove duplicates, and outputs preprocessed information to log files, including a preprocessing module that groups errors by type and provides notifications when error counts reach a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If error events are recorded in log files in real-time, then complete error information is captured, but log files roll over too quickly and become difficult to read
Solution Approach 1:
The patent applies preliminary action by buffering error events in memory before writing to log files. The buffer accumulates error events temporarily, allowing the system to preprocess and consolidate them before permanent storage, thereby preventing rapid log file rollover and improving readability while maintaining complete error information capture
Solution Approach 2:
The patent introduces an intermediary buffer between the error generation source and the log file storage. This buffer acts as a mediator that temporarily holds error events, enables processing and consolidation operations, and regulates the flow of data to log files, thus resolving the contradiction between complete capture and readability
2Loss of information
If all error events are logged individually, then complete error details are preserved, but storage space is consumed quickly and system performance degrades
Solution Approach 1:
The patent merges multiple error events of the same type into consolidated log entries. By grouping similar errors and recording them collectively rather than individually, the system preserves complete error information while significantly reducing the total number of log entries and storage space consumption
Solution Approach 2:
The buffer serves multiple functions: it stores error events temporarily, consolidates duplicate errors, manages write timing to log files, and controls storage consumption. This multi-functional approach allows the system to preserve error details while efficiently managing storage resources
3Productivity
If error logging is performed without buffering, then log files are updated immediately, but system performance is impacted and users remain unaware of errors
Solution Approach 1:
The patent implements periodic action by buffering error events and writing them to log files at intervals rather than immediately upon occurrence. This periodic flushing mechanism reduces the frequency of write operations, thereby improving system performance while the buffer ensures errors are not lost during the delay
Solution Approach 2:
The system provides feedback to users about buffered error events through notification mechanisms. When errors are accumulated in the buffer or when certain thresholds are reached, the system can alert users, ensuring timely awareness despite the delayed logging to files
Data Source
AI summary
A method for more effectively recording information in system logs is disclosed. In one embodiment, such a method includes detecting errors on a system such as a host system or storage system over a specified period of time. The method stores information associated with the errors in a memory buffer. The method further preprocesses the information in the memory buffer to condense the information and remove duplication. In certain embodiments, this preprocessing includes grouping errors by error type and providing a single stack trace or other information per error type. The method then outputs the preprocessed information to a log file. A corresponding system and computer program product are also disclosed.