Machine Learning Model for Unlabeled Security Log Parsing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face inefficiencies in parsing computer security data logs due to the need to maintain hundreds or thousands of parsers and employ a brute force method, which is computationally expensive and time-consuming, especially when the log type is unlabeled.
Innovation Solution
A machine learning model is trained to generate a probability distribution over possible data log types, allowing the system to select the most likely parser for an unlabeled data log, and if the initial selection fails, it can use the next highest probability parser, improving future predictions by learning from successful parses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a brute force method is used to parse computer security data logs, then all possible parsers can be tried to ensure correct parsing, but the computational cost and time consumption increase significantly
Solution Approach 1:
The system performs preliminary actions by training a machine learning model in advance to predict log types. When a new log arrives, the model quickly predicts the log type and selects the corresponding parser, avoiding the need to try all parsers. This preliminary prediction step significantly reduces parsing time while maintaining high accuracy.
Solution Approach 2:
The machine learning model acts as an intermediary between the incoming log and the parsers. Instead of directly trying multiple parsers on the log, the model first processes the log to predict its type, then selects the most appropriate parser. This intermediary step efficiently bridges the gap between unknown log types and suitable parsers.
2Adaptability or versatility
If hundreds or thousands of parsers are maintained to handle different log formats, then comprehensive log parsing capability is achieved, but system complexity increases
Solution Approach 1:
The machine learning model serves as a universal component that handles multiple functions: it classifies logs into types, predicts the most suitable parser, and adapts to new log formats through continuous learning. This single multi-functional model replaces the need to manually manage hundreds of specialized parsers, reducing system complexity while maintaining versatility.
Solution Approach 2:
Instead of maintaining actual copies of hundreds of parsers, the system uses the machine learning model to create a virtual representation of parser selection logic. The model learns patterns from training data and generates predictions that effectively replicate the decision-making process of choosing the right parser, without requiring physical copies of all possible parsers.
3Productivity
If a machine learning model is used to predict log types, then parser selection efficiency is improved, but the system requires training data and model training resources
Solution Approach 1:
The system performs preliminary training of the machine learning model using historical log data before deployment. This preliminary action prepares the model with knowledge of various log formats and patterns, enabling it to make accurate predictions quickly during operation. The training phase consolidates the data processing requirement into an upfront investment rather than ongoing operational overhead.
Data Source
AI summary
In some implementations, a method includes obtaining an unlabeled computer security data log and processing the unlabeled computer security data log using a machine learning model to generate a probability distribution that includes a respective probability for each of a plurality of possible log types. Each of the plurality of possible log types is associated with a corresponding parser that parses logs of the possible log type to extract structured computer security data. The method further includes selecting the possible log type having the highest probability and parsing the unlabeled computer security data log using the parser corresponding to the selected possible log type.