Entity Extraction Rule Recommender for SIEM Log Parsing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing entity extraction processes in Security Information and Event Management (SIEM) systems require significant time and uncommon expertise, making it difficult to adapt extraction rules to new data sources and users.
Innovation Solution
An entity extraction rule recommender system that automatically identifies and recommends suitable rules based on learning from user-defined mappings and applying them to new contexts, using a processor, memory, and a database of extraction rules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If entity extraction rules are manually created and adapted for each new data source, then extraction accuracy can be maintained, but the time and expertise required increase significantly
Solution Approach 1:
The system performs self-learning by automatically analyzing event data from new sources and generating extraction rules without human intervention. The machine learning model autonomously identifies patterns and creates rules, allowing the system to adapt to new data sources independently, thus eliminating the time-consuming manual rule creation process while maintaining accuracy through learned patterns
Solution Approach 2:
The system pre-processes and learns from baseline event data to build a foundation of extraction rules before encountering new data sources. By performing preliminary learning on representative data, the system prepares extraction patterns in advance that can be quickly adapted to new sources, reducing the time required for initial setup while maintaining accuracy through pre-established understanding of data structures
2Reliability
If manual expertise is required to adapt extraction rules, then rule quality can be ensured, but the complexity of operation increases
Solution Approach 1:
The system replaces manual expert operation with automated machine learning that independently analyzes data patterns and generates extraction rules. The model self-adjusts to new data sources by learning their specific structures and formats, ensuring rule quality through algorithmic consistency while eliminating the need for human expertise in rule creation and adaptation
Solution Approach 2:
The patent substitutes the mechanical process of manual rule creation and adaptation with an automated machine learning system. Instead of human experts manually analyzing and creating extraction rules, the system uses computational algorithms to automatically learn patterns from data and generate rules, replacing human cognitive work with automated processing that maintains reliability through consistent application of learning algorithms
3Manufacturing precision
If extraction rules are customized for specific contexts, then extraction effectiveness improves, but the difficulty of detecting and measuring appropriate rules increases
Solution Approach 1:
The system implements feedback loops where extraction results are continuously evaluated and used to refine extraction rules. The machine learning model learns from the effectiveness of previous extractions and automatically adjusts rules to improve performance on new data sources, making the process of identifying and measuring appropriate rules easier through iterative optimization based on performance feedback
Solution Approach 2:
The system autonomously identifies appropriate extraction rules for new data sources by analyzing their structures and patterns. The machine learning model self-determines which rules will be most effective for specific contexts without requiring human intervention to detect or measure appropriateness, automatically adapting rules to maximize extraction effectiveness for each unique data source
Data Source
Figure 1~2
Figure 3~5
Figure 6~7
AI summary
Security Information and Event Management tools, log management tools, log analysis tools, and other event data management tools are enhanced. Enhancements harvest entity extraction rules from queries, query results, and other examples involving the extraction of field values from large amounts of data, and help perform entity extraction efficiently. Entity extraction operations locate IP addresses, usernames, and other field values that are embedded in logs or data streams, for example, and populate object properties with extracted values. Previously used extraction rules are applied in new contexts with different users, different data sources, or both. An entity extraction rules database serves as a model that contains rules specifying parsing mechanisms. Parsing mechanisms may include regular expressions, separation character definitions, and may process particular file formats or object notation formats or markup language formats. A recommender suggests extraction rules to users, based on frequency, machine learning classifications, correctness certainty, or other considerations.