Dynamic Parsing Rule Reordering for Security Log Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity and volume of security logs from various applications and systems in large corporate organizations lead to inefficiencies in processing due to a static order of parsing rules, which does not adapt to changing log types and volumes.
Innovation Solution
A method and system that dynamically reorder parsing rules based on log type, volume, and complexity to optimize the routing of security logs to log normalization routines, using a processor to determine the most efficient order for processing incoming logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a static order of parsing rules is used, then the system is simple to implement, but processing efficiency deteriorates as log volume and complexity increase
Solution Approach 1:
The patent implements dynamic reordering of parsing rules based on real-time log type frequency and complexity metrics. The system continuously monitors log characteristics and adjusts the parsing rule sequence adaptively, transforming the static rule order into a dynamic configuration that optimizes processing efficiency for current log patterns while maintaining manageable system complexity through automated adjustment algorithms.
Solution Approach 2:
The system changes the parameters of parsing rule ordering based on log volume, type distribution, and complexity measurements. By adjusting the sequence parameters dynamically according to measured log characteristics, the system achieves improved processing efficiency without requiring complete system redesign, thereby managing complexity through parameter optimization rather than structural complexity.
2Measurement precision
If parsing rules are applied in a fixed sequence, then the system is easy to maintain, but the likelihood of correct log type identification decreases with diverse log types
Solution Approach 1:
The patent implements a feedback mechanism where the system monitors the performance of parsing rules in identifying log types and uses this information to dynamically adjust the rule sequence. By incorporating feedback from actual log processing outcomes, the system continuously improves identification accuracy for diverse log types while maintaining ease of operation through automated feedback-driven adjustments rather than manual reconfiguration.
Solution Approach 2:
The system performs preliminary analysis of log characteristics and pre-orders parsing rules based on anticipated log type distributions before actual processing occurs. This preliminary action allows the system to optimize the parsing rule sequence in advance for expected log patterns, improving identification accuracy while maintaining simplicity through pre-computed optimal sequences rather than complex real-time decision-making during processing.
3Measurement precision
If more parsing rules are applied to handle complex logs, then identification accuracy improves, but processing time increases
Solution Approach 1:
The patent dynamically adjusts the number and sequence of parsing rules applied based on real-time assessment of log complexity and type frequency. For common, simple log types, the system uses fewer parsing rules to minimize processing time, while for rare or complex log types, it activates additional rules to ensure accurate identification. This dynamic adaptation resolves the contradiction by making the rule application quantity variable rather than fixed.
Solution Approach 2:
The system applies a variable number of parsing rules based on log characteristics - using partial action (fewer rules) for straightforward log types where full rule application would be excessive and time-consuming, while applying excessive action (more rules) for complex or ambiguous log types where additional verification is necessary for accurate identification, thereby optimizing the balance between accuracy and processing time.
Data Source
AI summary
A system and a method for adjusting an order of parsing rules for routing incoming security logs to log normalization routines that are used for processing large volumes of security incident event data are provided. The method includes: sequentially applying, in a first order, a set of parsing rules to each of a first set of logs to determine destinations that correspond to log normalization routines for reformatting into a common format; determining whether an efficiency of the application of the parsing rules would be increased by reordering into a second order; and reordering the set of parsing rules into the second order based on the determination, so that the parsing rules will be sequentially applied to the next set of logs based on the second order thereof. The efficiency determination is based on volumes and complexities of various log types.