Log Message Tagging via Event-Type Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large distributed computing systems, the process of tagging log messages is computationally intensive and time-consuming, leading to increased storage costs and delays in real-time analysis due to the high volume and rate of log messages.
Innovation Solution
Implementing automated methods that collect blocks of log messages over time, perform event-type analysis, and apply tagging operations only when the event type matches qualified criteria, reducing unnecessary computational efforts and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a series of tagging operations are applied to each log message, then the log messages are properly tagged for analysis, but the computational cost and time increase significantly
Solution Approach 1:
The system performs preliminary event-type analysis on log messages before applying tagging operations. By categorizing log messages into event types first, the system determines which tagging operations are necessary, avoiding unnecessary tagging on messages that don't require it. This preliminary classification enables selective tagging that maintains accuracy while improving processing speed.
2Reliability
If log messages are buffered for tagging operations, then tagging can be applied systematically, but storage costs increase and real-time analysis is delayed
Solution Approach 1:
Instead of buffering all log messages for complete tagging processing, the system applies partial tagging actions selectively. By using event-type analysis to identify which messages need tagging and which don't, the system processes only the necessary subset of messages through the full tagging pipeline, reducing buffer requirements and enabling faster real-time analysis while maintaining tagging completeness for relevant messages.
3Reliability
If tagging operations are applied to all log messages, then comprehensive tagging coverage is achieved, but unnecessary computational resources are consumed
Solution Approach 1:
The system applies different tagging strategies to different log messages based on their event types. Instead of uniformly applying all tagging operations to every message, the system tailors the tagging approach locally to each message's characteristics. This selective application ensures comprehensive tagging coverage for messages that need it while avoiding unnecessary computational resource consumption on messages that don't require extensive tagging.
Data Source
AI summary
This disclosure is directed to tagging tokens or sequences of tokens in log messages generated by a logging source. Event types of log messages in a block of log messages are collected. A series of tagging operations are applied to each log message in the block. For each tagging operation, event types that are qualified to receive the corresponding tag are identified. When a log message is received, the event type is determined and compared with the event types of the block in order to identify a matching event type. The series of tagging operations are applied to the log message to generate a tagged log message with the restriction that each tagging operation only applies a tag to token or sequences of tokens when the event type is qualified to receive the tag. The tagged log message is stored in a data-storage device.