Intelligent Log Filter for System Message Utility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large system logs overwhelm resources with excessive data, leading to increased costs in collection, monitoring, transmission, and storage, as well as bandwidth consumption, and existing solutions fail to effectively reduce log size by merely changing the viewing experience or removing redundant messages without addressing the value of non-redundant messages.
Innovation Solution
An intelligent unsupervised log filter determines the informational value of system log messages and deletes those of low utility value, reducing the data stored in system logs based on their utility, using a sliding window context to assess predictability and assign utility scores compared to a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If system logs collect all messages from system modules, then complete information is available for analysis, but log size increases causing resource overload and increased costs
Solution Approach 1:
The patent extracts only the most important log messages by identifying and removing low-utility messages. The system uses predictive models to determine which messages provide valuable information and which are redundant or predictable, thereby extracting only the necessary subset of messages for analysis while discarding the rest.
Solution Approach 2:
The patent changes the parameter of message utility by introducing a utility score based on predictability. Messages are evaluated based on how predictable they are given previous messages, and this predictability parameter is used to filter and select only the non-predictable (high utility) messages for retention.
2Loss of information
If system logs store all messages in buffers, then complete log data is available, but buffer overflow occurs leading to message dropping
Solution Approach 1:
The patent applies preliminary action by filtering messages before they are stored in the buffer. The system evaluates each message's utility in advance using predictive models and only stores high-utility messages in the buffer, preventing overflow before it occurs rather than dealing with overflow after messages are dropped.
Solution Approach 2:
The system extracts only essential messages for buffer storage by removing low-utility messages beforehand. This extraction process ensures that the buffer contains only the most important messages, maximizing the use of limited buffer space while preventing overflow.
3Loss of information
If system logs transmit all collected logs to central repository, then complete data is available for monitoring, but network bandwidth is consumed excessively
Solution Approach 1:
The patent extracts only high-utility messages for transmission to the central repository. By using predictive models to identify and remove low-utility messages locally before transmission, the system significantly reduces the amount of data sent over the network while maintaining the completeness of important information.
Solution Approach 2:
The system performs preliminary filtering and utility assessment before transmission occurs. Messages are evaluated and selected for transmission in advance, ensuring that only the most important data is sent over the network, thereby reducing bandwidth consumption while maintaining data completeness for monitoring purposes.
4Loss of information
If users manually inspect all system log messages, then complete analysis is possible, but time and human resources are consumed excessively
Solution Approach 1:
The patent extracts only the most important messages for manual inspection. By using predictive models to identify and retain only high-utility messages, the system reduces the volume of data that requires human attention while maintaining the completeness of analysis for critical issues.
Solution Approach 2:
The system performs preliminary filtering to separate high-utility from low-utility messages before they reach the user. This preliminary action ensures that users only receive and inspect the most important messages, significantly reducing manual inspection time while maintaining complete analysis capability for critical events.
Data Source
AI summary
Systems and methods disclosed herein provide intelligent filtering of system log messages having low utility value. In providing the filtering, the systems and methods determine the utility value of a system log message and delete the message from the system log if the message is determined to be of low utility value. As such, embodiments herein provide an system log filter, which reduces the amount of data stored in the system log based on the utility value of the message.