ML-Assisted Syslog Parser Generation for Diverse Log Schemas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprises lack the resources to process log files from various formats and standards, as existing parsers are limited to specific formats, making it difficult to detect cybersecurity threats effectively.
Innovation Solution
A system using machine learning techniques to generate parsers that can classify and map log files to a schema, allowing for the creation of parsers that can handle diverse formats and standards, and enabling users to edit and improve the parsers for better accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a parser is designed to process log files following a particular schema, then the parser can accurately process log files of that format, but the parser is unable to process log files that deviate in format from that which the parser is designed to process
Solution Approach 1:
The patent creates a universal parser generation system that can handle multiple log file formats and schemas. Instead of designing separate parsers for each format, the system generates parsers dynamically based on the input log file's schema, enabling a single system to perform multiple parsing functions across diverse formats including syslog, CEF, and proprietary formats.
Solution Approach 2:
The system changes the parameters of the parser based on the log file schema being processed. By analyzing the schema of incoming log files and dynamically adjusting parser configuration parameters, the system adapts to different formats while maintaining accurate parsing. This allows the same parser generation mechanism to work across multiple schemas by modifying its operational parameters.
2Productivity
If enterprises contract out cybersecurity threat detection to specialized enterprises, then resource constraints are overcome, but challenges arise in detecting threats from enterprises that collect data using unsupported or proprietary devices and standards
Solution Approach 1:
The system employs self-service by automatically analyzing incoming log files, identifying their schema format, and generating appropriate parsers without manual intervention. This automation enables the specialized enterprise to handle diverse proprietary and unsupported formats efficiently, overcoming resource constraints while maintaining broad format support for comprehensive threat detection.
Solution Approach 2:
The system performs preliminary actions by pre-processing and analyzing log file schemas to generate appropriate parsers before actual threat detection begins. This preliminary schema analysis and parser generation enables the system to handle diverse formats efficiently during the main threat detection process, supporting both productivity and adaptability.
3Loss of information
If data elements in log files from different standards are processed, then comprehensive threat detection is possible, but the same or similar data elements may have different meanings in different log files
Solution Approach 1:
The patent introduces an intermediary schema mapping layer that translates data elements from various log file standards into a unified internal representation. This intermediary layer preserves the original meaning of data elements by maintaining schema-specific mappings while enabling consistent threat detection across different formats, reducing the complexity of direct multi-format processing.
Solution Approach 2:
The system segments the parsing and processing workflow into distinct phases: schema identification, parser generation, data extraction, and threat analysis. This segmentation allows each phase to handle format-specific complexities independently, preserving data element meanings while managing overall system complexity through modular processing steps.
Data Source
AI summary
Systems and methods for generating a parser from a log file including: receiving a log file, wherein the log file is a structured text file of a plurality of data elements; invoking a machine learning model to: process the log file to identify name-value-pairs from the data elements; classify the log file as being associated with a schema based in part on the name-value pairs; map a first name-value pair to a first input field of the schema based on characteristics of the first name-value pair; determine a confidence level associated with mapping the first name-value pair to the first input field; and when the confidence level for mapping the first name-value pair exceeds a threshold, provide the first name-value pair to the first input field; and generating a parser from the plurality of input fields of the schema.