Naïve Bayesian Probabilistic Mapper for Unstructured Log Parsing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in automatically extracting information from unstructured log data and mapping it to a structured schema, especially when the format is unknown or contains free-form plain text, as deterministic parsers are ineffective in such cases.
Innovation Solution
An unstructured event parser is employed, which uses a probabilistic mapper based on the Naïve Bayesian probability model to analyze events, tokenize them, capture information, and map tokens to structured fields, generating a structured event through tokenization, information capture, and mapping phases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a deterministic parser is used to map event information to structured fields, then the parsing process is simple and reliable for known formats, but it cannot handle unstructured events with unknown or free-form text formats
Solution Approach 1:
The patent changes the fundamental parameter of the parsing approach from deterministic to probabilistic. Instead of using fixed parsing rules that work only for known formats, the system uses a probabilistic mapper that assigns likelihood scores to different field mappings based on token characteristics and training data. This parameter change enables the parser to handle unstructured events while maintaining structured output.
Solution Approach 2:
The patent replaces the mechanical deterministic parsing system with a statistical probabilistic system. The probabilistic mapper uses machine learning models trained on historical event data to determine field mappings, substituting rigid mechanical parsing rules with flexible statistical inference that can adapt to unknown formats.
2Adaptability or versatility
If probabilistic mapping is used to handle unstructured events, then adaptability to unknown formats is improved, but the mapping process becomes non-deterministic and more complex
Solution Approach 1:
The system incorporates feedback mechanisms where the probabilistic mapper continuously refines its field mapping probabilities based on the input event context and historical data. The training process uses feedback from labeled event data to improve the accuracy of probability calculations, enabling consistent reliable mappings despite the non-deterministic nature of probabilistic processing.
Data Source
AI summary
An “unstructured event parser” analyzes an event that is in unstructured form and generates an event that is in structured form. A mapping phase determines, for a given event token, possible fields of the structured event schema to which the token could be mapped and the probabilities that the token should be mapped to those fields. Particular tokens are then mapped to particular fields of the structured event schema. By using the Naïve Bayesian probability model, a “probabilistic mapper” determines, for a particular token and a particular field, the probability that that token maps to that field. The probabilistic mapper can also be used in a “regular expression creator” that generates a regex that matches an unstructured event and a “parameter file creator” that helps a user create a parameter file for use with a parameterized normalized event generator to generate a normalized event based on an unstructured event.