Event Message Sampling Policy for Log Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large distributed computing systems, managing and analyzing vast volumes of event messages is challenging due to their sheer size and complexity, making it difficult for administrators to identify relevant information for operational anomaly detection, system maintenance, and security breaches.
Innovation Solution
Implementing a method to sample event messages based on predefined policies, where event messages are stored in a separate sample log file, allowing for selective storage and display in a graphical user interface, enabling users to change policies and sample rates for different event types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all event messages are stored in the event log, then complete information is preserved for analysis, but storage requirements and query complexity increase significantly
Solution Approach 1:
The patent segments the event log into two distinct components: a sample log that stores only sampled event messages according to configurable policies, and a complete event log that retains all messages. This segmentation allows the system to maintain a lightweight, quickly queryable sample log for routine monitoring while preserving the complete log for comprehensive analysis when needed, thus resolving the contradiction between information completeness and storage efficiency.
Solution Approach 2:
The patent extracts a representative subset of event messages from the complete event log based on configurable sampling policies (e.g., random sampling, sampling by event type, sampling by frequency). This extracted sample log contains only the most relevant or representative messages, reducing storage requirements and query complexity while still providing sufficient information for operational anomaly detection and system monitoring.
2Reliability
If all event messages are stored and analyzed, then comprehensive monitoring is achieved, but query efficiency and analysis speed decrease
Solution Approach 1:
The patent creates a separate sample log segmented from the complete event log, containing only sampled messages based on configurable policies. This segmentation enables fast queries on the small sample log for routine monitoring tasks, while the complete log remains available for comprehensive analysis when needed, thus resolving the contradiction between monitoring comprehensiveness and query efficiency.
Solution Approach 2:
The patent applies partial action by storing and analyzing only a representative sample of event messages rather than all messages. The sampling policies (random sampling, sampling by event type, sampling by frequency) ensure that the sample log contains sufficient information for effective monitoring while reducing the volume of data requiring continuous analysis, thereby improving query efficiency without sacrificing monitoring reliability.
3Device complexity
If event messages are stored in a single log file, then simplicity is maintained, but flexibility in sampling and policy management is reduced
Solution Approach 1:
The patent segments the logging system into distinct components: a sample log for storing sampled messages and a complete event log for storing all messages. This segmentation provides the flexibility to apply different sampling policies (random sampling, sampling by event type, sampling by frequency) and configure different retention strategies for different log types, while maintaining relatively simple structures for each individual log.
4Productivity
If a small sample log is maintained, then storage requirements and query time are reduced, but information completeness may be compromised
Solution Approach 1:
The patent extracts a representative subset of event messages using configurable sampling policies that ensure the sample log contains sufficient information for effective monitoring. The policies include random sampling, sampling by event type (e.g., storing all error messages), and sampling by frequency (e.g., storing the most frequent event types). This extraction strategy maintains information completeness for critical events while reducing overall data volume, thus resolving the contradiction between analysis speed and information completeness.
Solution Approach 2:
The patent changes the sampling parameters dynamically based on system needs and event types. Different sampling rates and policies can be applied to different event types, allowing the system to adjust the balance between sample log size and information completeness. For example, critical events like errors may be sampled at 100% rate while routine events use lower sampling rates, maintaining analytical value while controlling storage requirements.
Data Source
AI summary
Methods and systems to sample event messages are described. As event messages are generated by one or more sources, the event messages are stored in a storage queue. An event message policy that represents conditions for storing event messages in a sample log file are input. For each event message output from the storage queue, the event message may be stored in a sample log file when one or more of the conditions of the event message policy are satisfied. The event messages of the sample log file may be displayed in a graphical user interface that enables a user to change the event message policy.