Log Message Categorization Using Discriminatory Token Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for categorizing log messages into categories are either expensive and not scalable due to the need for exact knowledge of log message specifications or become overly complex when trying to accommodate a large number of categories, making them difficult to maintain and adapt to new patterns.
Innovation Solution
A method that involves analyzing computerized messages to identify discriminatory tokens and determine their weights, then classifying messages based on similarity to categories using a processor-coupled device, which simplifies the process and allows for efficient adaptation to new devices and categories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If customized parsers are used to process log messages from each device, then log message categorization accuracy is improved, but system complexity and cost increase linearly with the number of devices
Solution Approach 1:
The patent creates a universal parser that can handle log messages from multiple different devices and vendors using a single standardized interface. The system defines a common log message format with standardized fields that can accommodate data from various sources, eliminating the need for device-specific parsers while maintaining categorization accuracy through a unified processing approach.
Solution Approach 2:
The patent transforms the approach by changing from device-specific parsing parameters to standardized field parameters. Instead of creating custom parsers with device-specific logic, the system uses a standardized set of fields (timestamp, severity, source, message, etc.) that can be populated by different devices, reducing system complexity while maintaining the ability to accurately categorize logs from multiple sources.
2Adaptability or versatility
If log message specifications are changed by vendors, then device functionality is improved, but parser reliability deteriorates as parsers must be continuously updated
Solution Approach 1:
The patent implements a dynamic field structure where the log message format can adapt to new device requirements while maintaining backward compatibility. The standardized format includes extensible fields that allow vendors to add new data elements without breaking existing parsers. The system can dynamically handle both fixed and variable fields, allowing the specification to evolve while maintaining parser reliability through a flexible, future-proof architecture.
3Adaptability or versatility
If rules and regular expressions are used to match log messages to categories, then categorization coverage is improved, but system complexity increases making maintenance difficult
Solution Approach 1:
The patent segments the log message into distinct standardized fields (timestamp, severity, source, message, etc.) that can be independently processed. Instead of using complex regular expressions to extract information from unstructured text, the system parses the message into structured fields that can be easily matched against categorization rules. This segmentation simplifies the matching process while maintaining comprehensive categorization coverage through structured field comparison.
Data Source
AI summary
There is disclosed herein techniques for categorizing computerized messages into categories. In one embodiment, there is disclosed a method. The method comprising performing an analysis of one or more computerized messages that includes identifying a set of discriminatory tokens in the one or more computerized messages that are representative of a category and determining for each discriminatory token a respective weight by which the token describes the category. The method also comprises determining a similarity between a computerized message and the category based on the content of the computerized message, the set of discriminatory tokens and the respective weights. The method further comprises classifying the computerized message as belonging to the category upon determining that the computerized message and the category are similar.