Adaptive Log Parsing with Probabilistic NER
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managed security services providers face challenges in efficiently normalizing and analyzing diverse security logs from various clients and devices, leading to increased costs and vulnerabilities due to the need for extensive manual scripting and potential delays in updating parsing scripts, which can result in missed security threats.
Innovation Solution
A security system utilizing adaptive parsing engines that employ probabilistic recognition and Named Entity Recognition (NER) to dynamically normalize and curate unstructured log data into a machine-readable format, enabling automatic analysis and alert generation without requiring new parsing scripts, even for unrecognized log formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rule-based parsing scripts are used to normalize logs, then normalization accuracy is improved, but the system requires extensive manual scripting and cannot adapt to new log formats, increasing operational complexity and time delays
Solution Approach 1:
The system employs machine learning models that automatically learn and adapt to new log formats without requiring manual intervention. The models self-train on incoming log data, automatically updating their parsing capabilities when encountering new formats, thereby eliminating the need for manual script creation and maintenance while maintaining high normalization accuracy.
Solution Approach 2:
The system transitions from static rule-based parameters to dynamic machine learning models that can adjust their parameters based on learned patterns. This allows the system to adapt to varying log formats by changing its internal parameters through learning rather than through manual reconfiguration, reducing operational complexity while preserving accuracy.
2Adaptability or versatility
If manual scripting is used to update parsing scripts for new log formats, then adaptability to new formats is improved, but response time deteriorates due to the time required to create and deploy new scripts
Solution Approach 1:
The machine learning models automatically detect and adapt to new log formats in real-time without waiting for manual script updates. When new log formats are encountered, the models self-train on the incoming data and immediately begin parsing them correctly, eliminating the time delay associated with manual script creation and deployment while maintaining full adaptability.
Solution Approach 2:
The system performs preliminary learning and adaptation in real-time as logs arrive, rather than waiting for batch updates. The machine learning models continuously learn from incoming data, preparing parsing capabilities in advance before they are needed, which eliminates response time delays when new formats are encountered.
3Productivity
If extensive data engineering teams are deployed to manage parsing scripts, then normalization capability is improved, but operational cost increases significantly
Solution Approach 1:
The machine learning models perform the normalization function autonomously without requiring human operators to create or manage parsing scripts. The models self-train, self-update, and self-maintain their parsing capabilities, replacing the need for extensive data engineering teams while maintaining or improving normalization capability, thereby significantly reducing operational costs.
Solution Approach 2:
The system replaces the mechanical process of manual script creation and management with automated machine learning-based parsing. This substitution eliminates the need for human labor in script development while maintaining high normalization capability, leading to significant cost reductions without sacrificing productivity.
4Quantity of substance
If parsing scripts are not updated promptly for new log formats, then operational cost is reduced, but security reliability deteriorates due to missed security events
Solution Approach 1:
The machine learning models continuously monitor incoming logs and automatically adapt to new formats in real-time, ensuring that security events are never missed due to outdated parsing capabilities. This self-updating mechanism maintains high security reliability without requiring costly frequent script updates, as the models learn and adapt autonomously as new log formats appear.
Data Source
AI summary
A method of normalizing security log data can include receiving one or more security logs including unstructured data from a plurality of devices and reviewing unstructured data of the one or more security logs. The method also can include automatically applying a probabilistic model of one or more engines to identify one or more attributes or features of the unstructured data, and determine whether the identified attributes or features are indicative of identifiable entities, and tagging one or more identifiable entities of the identifiable entities, as well as organizing tagged entities into one or more normalized logs having a readable format with a prescribed schema. In addition, the method can include reviewing the one or more normalized logs for potential security events.